This reminds me of themes I recently saw in [Harness Engineering is not Enough: Why Software Factories Fail](https://www.youtube.com/watch?v=Ib5GBkD555M) (Warning: the last 3 slides seem like an advertisement). One thing I liked is how Dex has a little graphic he glossed over showing software development is
- 25% planning & aligning with other teams
- 25% coding
- 25% testing/verifying
- 25% code review/rework
One argument was that agentic coding speeds up that coding part a bunch. So maybe there's 2x speedup in coding. But that's only a small speedup in the totality of everything software engineers do.
AI is great because my job is a lot easier, but it produces some extremely crappy code that would've NEVER passed a code review 1 year ago.
You could argue that it speeds up development by 5X or more, but then it slows down testing/verifying, code review, and in many cases it makes it impossible to review/rework by hand.
I just do what I'm told at work but even though I'm sure I'm in the minority I'm extremely skeptical that LLMs can produce any good-quality code.
Looks to me like we've just lowered the bar--by a lot--and stopped looking at the code that goes out. Every time I look into how Claude implemented something it's completely insane, with no way to refactor it or maintain it in the future.
I read this a lot, do you have examples of this? Last year the code produced by LLMs was pretty sloppy, but recent SOTA models seems pretty good to me. I'm curious if I'm missing something or it's folks using different models or difficulty levels.
> it produces some extremely crappy code that would've NEVER passed a code review 1 year ago
The 10x speed up comes from the fact that we have all given up on properly reviewing each others code and we just say “meh, it will be fine, lgtm”
Yes, exactly my point.
BUT, to me if we hadn't lowered the bar MASSIVELY, it would NOT be fine.
I've seen people at my work that will push or even deploy features that don't do what their main purpose is and they just dismiss the issue saying it's a bug. As an example, a button that's supposed to open a modal window if you click on it. If you go test it and the modal doesn't open, they'll say "oh, there's a bug".
You developed or even deployed something that is wrong and completely useless. Just one year ago, this would be unheard of and you would've gotten a really hard time for it. Now it's just a bug and part of how we're doing things.
Like I said, I just do what I'm told at work but to me this is just wild. There's no way it won't come back to bite us.
This. The bar has never been lower.
> So maybe there's 2x speedup in coding. But that's only a small speedup in the totality of everything software engineers do.
Amdahl's Law should be familiar to anyone with a 4y computer science/engineering degree. Why aren't they applying it to their own throughput?
Well, there’s probably several reasons:
I’m sure others as well.Let's apply it:
- 25% planning & aligning with other teams
- 25% coding
- 25% testing/verifying
- 25% code review/rework
I'd say that thanks to LLM assistance I'm 10x faster at coding, code review/rework, and testing/verifying. (LLMs can partially automate testing/verifying too, and the code is higher quality now as well so less testing/verifying is necessary).
So that leaves us with:
- 92.5% planning & aligning with other teams
- 2.5% coding
- 2.5% testing/verifying
- 2.5% code review/rework
Obviously, that makes zero sense as a split. If you saw any organization doing that, you'd suggest having fewer teams, more silos, etc. Maybe you have designers produce code, instead of showing the designs to coders and having the coders implement it. Maybe you force all your engineers to dogfood the product that way they can identify issues themselves rather than needing QA teams to do it. And so on. so the last category, "planning & aligning with other teams", falls too.
The other hard to measure part is that I see/hear a lot of LLM usage going towards dev work that was never prioritized before.
Suddenly devs who were cranking out features with no interest in infrastructure are attacking giant refactors to make the code more understandable to the LLM.
Other devs are using LLMs to build themselves quality of life SDLC tools completely separate from the core code base.
Plenty of other examples of this.
The quality of my bash scripts is 100x what they used to be. Proper help messages, flag parsing, functions, internal variables, using sed all the time (I could never wrap my head around that).
Of course, the main issue is that they’re completely undebbugable now. My bash scripts used to be a sequential list of commands, now they’re 500 lines of variable laden functions.
Is my life any better? Dunno. But it’s satisfying (until there’s a bug)
> the main issue is that they’re completely undebbugable now
Well, that means the quality actually dropped then :). Looking impressive isn't equal to quality, understandability and reliability is
Not necessarily…. As with all engineering choices there are tradeoffs involved. There can be both advantages and disadvantages to using AI to build scripts. And more likely than not you’ll need AI to help debug the scripts when they have issues. Real tradeoffs to consider.
100x UX quality, not DX quality.
I switched to js for dev related scripting after seeing how much python scripting is used for portable c++ projects like chromium.
A node glob() or a regexp string.replace call is probably easier to read than spaghetti shell.
And your llm might do a much better job of creating clean, readable and testable code.
Just one more data point but this is what I’m seeing too. The extreme ‘ruthless’ prioritization has stopped and devs (who want it) have breathing room to make the enhancements they’ve been wanting all along. Turns out what is good for AI is also good for regular old school SDLC. All you have to do is say it will make AI more effective.
I use CC to do the testing and verifying in multiple deployment environments. It’s faster at things like coming up a plausible test payload or ten, querying a bunch of metrics from datadog, querying and verifying state from a db before and after testing, scanning all the deployed logs for relevant errors. For me that has been more of an uplift than the coding. I still feel like communication it’s worst at. So writing tickets, getting a decision out of a group, documenting only the important things, etc. it’s still not that good at
Agentic tools changed the workflow in our org, we were very tense before with a simple yet careful team process.. Since last winter we're now pushing a lot more but the teamwork (which was brittle before) is now mostly gone, everybody can roll on its own, but the review process didn't scale and now monthly deliveries are full of "seems to work". nobody checks properly, nobody reviews, nobody tighten any bolts.. and nobody cares much anyway, the system allows it.