When anecdote and data don't align, it's usually the data that's wrong.
Not always the case, but whenever I read about these strained studies or arguments about how AI is actually making people less productive, I can't help but wonder why nearly every programmer I know, myself included, finds value in these tools. I wonder if the same thing happened with higher level programming languages where people argued, you may THINK not managing your own garbage collector will lead to more productivity but actually...
Even if we weren't more "productive", millions prefer to use these tools, so it has to count for something. And I don't need a "study" to tell me that
TFA says clearly that it is likely that AI will make more productive anyone working on an unfamiliar code base, but make less productive those working on a project they understand well, and it gives reasonable arguments for why this is likely to happen.
Moreover, it acknowledges that for programmers working in most companies the first case is much more frequent.
I have written every line of code in the code base I mostly work in and I still find it incredibly valuable. Millions use these tools and a large percentage of them find them useful in their familiar code base.
Again, overwhelming anecdote and millions of users > "study"
> Interestingly the developers predict that AI will make them faster, and continue to believe that it did make them faster, even after completing the task slower than they otherwise would!
In this case clearly anecdotes are not enough. If that quote from the article is accurate, it shows that you cannot trust the developers time perception.
I agree, its only one study and we should not take it as the final answer. It definitely justifies doing a few follow up evaluations to see if this
> If that quote from the article is accurate, it shows that you cannot trust the developers time perception.
The scientific method goes right out the window when it comes to true believers. It reminds me of weed-smokers who insist getting high makes them deep-thinkers: it feels that way in the moment, but if you've ever been a sober person caught up in a "deep" discussion among people high on THC, oh boy...
Or I cannot trust a contrived laboratory setting with it's garden of forking paths.
https://mleverything.substack.com/p/garden-of-forking-paths-...
I did not say to trust it. I do not need to trust it.
If I run my own tests on my own codebase I will definitely use some objective time measurement method and a subjective one. I really want to know if there is a big difference.
I really wonder if its just the individuals bias showing. If you are pro-AI you might overestimate one, and if you are against it you might under-estimate it.
That's fair, I agree.
> I can't help but wonder why nearly every programmer I know, myself included, finds value in these tools.
One of the more interesting findings of the study mentioned was that the LLM users, even where use of an LLM had apparently degraded their performance, tended to believe it had enhanced it. Anecdote is a _really_ bad argument against data that shows a _perception_ problem.
> Even if we weren't more "productive", millions prefer to use these tools, so it has to count for something.
I mean, on that basis, so does homeopathy.
Like, it's just one study. It's not the last word. But "my anecdotes disprove it" probably isn't a _terribly_ helpful approach.
Also, "anecdotes > data" as a general heuristic is a red flag. But like if clowns had a country and their flag were red. That kind.