Part of the explanation (for me) is that the slow down comes later, and that part of the cost is externalized to team members. If you go all-in agentic on a project that has at least reasonable code quality, you can certainly gain a temporary boost by stopping to read the code for a while. At least long enough to get yourself and the rest of the team hooked on the idea.
Kinda reminds me of the "beginner's luck" problem for gamblers.
Also my experience, it works somewhat ok on large code bases that I designed and built myself before but after months of agentic development they sure start to degrade. I think if you start from scratch with agentic development there is no foundation for the models to anchor to.
I suspect this is just because of context. The AIs have very limited context related to us. I suspect that this is partially a result of the AIs being forced to be highly generalized machines which will work out of the box with anyone's prompt so they can't afford to be tailored to a particular contextual pattern. This lack of a robust pre-defined contextual framework that is relevant to the human who is prompting them, coupled with limited inputs to what the human actually wants and sees, result in context drift as the agent continues to go down its own path.