It depends a lot on how you use it, and how much effort you put into getting a knack for using it (which is rough because you're always worried that knack might be out of date within a month or two).
I use Claude, Codex, and the Gemini CLI (all "supervised command line agents"). I write Go. I am certain that agents improve my productivity in a couple common scenarios:
1. Unsticking myself from stalling at the start of some big "lift" (like starting a new project, adding a major feature that will pull in some new dependency). The LLM can get things very wrong (this happens to me maybe 20% of the time), but that doesn't matter, because for me the motion of taking something wrong and making it "righter" is much less effort than getting past a blank page, assembling all the resources I need to know how to hello-world a new dependency, that kind of thing. Along with "wrestling for weeks with timing-dependent bugs", this kind of inertia is one of like two principal components of "effort" in programming for me, so clearing it is a very big deal. I'm left at a point where I'm actually jazzed about taking the wheel and delivering the functionality myself.
2. Large mechanical changes (like moving an interface from one component to another, or major non-architectural refactors, or instrumentation). Things where there's one meta-solution and it's going to be repeated many times across a codebase. Easy to review, a lot of typing, no more kidding myself for 20 minutes that I can make just the right Emacs macro to pull it off.
3. Bug hunting. To be clear: I am talking here about code I wrote, not the LLM. I run it, it does something stupid. My first instinct now is to drop into Claude or Gemini, paste the logs and an @-reference to a .go file as a starting point, and just say "wtf". My hit rate on it spotting things is very good. If the hit rate was even "meh borderline" that would be a huge win for the amount of effort it takes, but it isn't, it's much better.
I'm guessing a lot of people do not have these 3 experiences with LLM agents. I'm sorry! But I do think that if you stipulate that I'm not just making this up, it's hard to go from here to "I'm kidding myself about the value this is providing". Note that these are three cases that look nothing at all like vibe-coding.
This matches my experience exactly. #3 is the one I've found most surprising, and it can work outside the context of just analyzing your own code. For example I found a case where an automated system we use started failing due to syntax changes, despite no code changes on our part. I gave Claude the error message and the context that we had made no code changes, and it immediately and correctly identified the root cause as a version bump from an unpinned dependency (whoops) that introduced breaking syntax changes. The version bump had happened four hours prior.
Could I have found this bug as quickly as Claude? Sure, in retrospect the cause seems quite obvious. But I could just as easily rabbit holed myself looking somewhere else, or taken a while to figure out exactly which dependency caused the issue.
It's definitely the case that you cannot blindly accept the LLM's output, you have to treat it as a partner and often guide it towards better solutions. But it absolutely can improve your productivity.
My experience with getting it to write code has not been good so far. Today I had a pretty mechanical task in Go. Basically some but not all functions from a package moved into another package so I asked Gemini to just change instances of pkg.Foo to pkgnew.Foo and import pkgnew in those files. It just got stuck on one file for 2 minutes and at that point I was already halfway through find/replacing it on my own.
For me it's been somewhat useful to ask questions but always fails at modifying any code in a big codebase.
With LLM assistance I managed to pinpoint an esoteric Unity issue to within 5 lines of code.
I've had one 3-day basic course in Unity, but I know how to prompt and guide an AI.
I agree with these points and would add another: writing code that I don't strictly need and would normally avoid writing at all simply because it's easier to take the lazy route and do without it. This morning while on the shitter, I had claude code use dbus to interface with tumbler for generating thumbnails. My application doesn't neeed thumbnails and normally my reaction would be "Dbus? Eww. I'll just do that later (never)" But five minutes of Claude Code churning got the job done.
> 3. Bug hunting.
Agreed, and also configuration debugging. Treat the LLM as interactive documentation for libraries, frameworks, etc. that you may be using. They're great for solving problems in getting systems up and running, and they explain things better than the documentation because it's specific to your scenario.