Yes this is true to some extent. I've been using LLMs to run some computer vision tests, and I've certainly noticed myself running into the trap of "just one more AI experiment", or "just one more change" while neglecting to actually properly integrate the learnings into my mental model.

However, without an agent running its own experiments on a cloud GPU, would I realistically have invested my limited work hours and tried evaluating 10 different models, each with 10 different tuned parameters, to solve my specific use case?

Or would I have tried 1-2 models and spent my time trying to optimize those models?

I think there is some merit to the spray and pray approach when one is in the exploration phase of the solution space.

Also, on more than one occasion now, I have had fable halve the inference latency of a model simply because the original implementation from an academic included unnecessary GPU-to-CPU-to-GPU transfers or similarly inefficient operations. Those optimizations came at essentially 0 time cost to me and I can verify that the outputs are byte-identical. Pretty sweet!