This sort of thing makes me think that if I were engineering a biome, this sort of specialized spider would be exactly how I would solve the problem of a species that I accidentally made too prolific. The spider is a targeted fix: only preys on the ants, and if the ant problem ever resolves itself, the spider goes away.

I think this is a case where the theory of evolution feels pretty handy-wavy and doesn't answer the mechanism of how this happens. The article even talks about "bioengineered" silk! There are certainly lots of cases of evolution, so it is clear that things do evolve, but the problems of a theory are revealed in the edge cases, and I think this spider suggests that the theory really needs a lot of work to be a robust theory. I don't think a simple gradient descent (evolving toward more fitness) has explanatory power in this case. A theory that hand-waved around edge cases like this would never fly in Physics; at best it would be considered a holding theory until a better one could be found, like dark matter.

(Plus, I think it's fun to think about bioengineering in a sci-fi, or even Babylon 5 way)

Couple things here:

First, ecosystems don't work on this concept of "balance" or "problems to fix". There are ecological niches that are exploited by a species or not. We have both generalists and specialists, predators that will eat anything and predators that have specialized in a particular prey or hunting approach. The world is full of organisms with remarkably complex behaviors, the field of evolutionary biology would collapse if this sort of behavior disproved it.

Second, there's really nothing hand-wavy about how specialized behaviors evolve, and in fact there is a mountain of research in the fields of evolutionary biology and behavioral ecology that have significant explanatory power here. I would argue that you're the one being hand-wavy saying "I don't think a simple gradient descent...". Evolution is not a simple gradient descent, and no evolutionary biologist would ever argue that it is.

It might be more helpful to think of it as small changes in “genome-vector” taking place across generations of the species with the filter of it being not bad enough to cause extinction of the bespoke variation, as opposed to evolving towards more fitness as an optimisation objective. When thought of like this, one can imagine how high-dimensional the “feature”-space becomes, which leads to such intricate engineering we see in nature.

There's also typically a ton of standing genetic variation that is relatively fitness-neutral, but can then lead to more rapid or directional evolution if conditions change. For example, standing variation in silk elasticity may have been more fitness neutral when capturing other prey, but after encountering this ant species, spiders with more elastic silk were more successful and had a fitness advantage.