One thing Dario said is important reflecting on (but from another angle): where's the big deliverable from AI? If AI makes us 10x more productive, the 3 years since its popularization were enough for a product that would have taken 30 years to build without AI, for instance.
I'm an AI advocate but that question makes me feel that AI is simply "very useful" rather than being a historical game changer for humanity.
Anyone saying 10x as a serious claim is clearly using a round number and vibes; however, even if it were so, AI getting popular 3 years ago does not mean what you say.
The earliest used models in this category would easily 10x (what did I just say) the creation of one-off short scripts, but you're not writing a 30-year app out of just a bunch of short scripts.
The METR time horizons graph suggests we can now get 10x (ahem) speedup on solving most coding problems that take us a few hours and about half of problems that take 2 days. Amdahl's law bites: even infinity speedup on half your problems is only 2x overall.
If you let an LLM loose, with a huge budget, what's the biggest artefact it can make before it drowns under the weight of bad decisions? The C compiler and web browser headlines a while back? The maths papers we see now that solve problems which stumped the maths world for decades?
But this is the other side of the same coin: teamwork. One person getting a thing made in twelve months vs a team of a hundred, you can scale up fast with money when you have a proof of concept, and an LLM can make a lot of proofs of a lot of concepts even in free accounts.
LLM-assisted software engineering seems to be very efficient if you have a deterministic target. (bun rewrite from zig to Rust, 100% Node.js compatibility, pnpm compatibility -- https://github.com/oven-sh/bun/pull/38333)
Problem with that is that it's not proper rust code, it's some kind of franken zig style rust port.
The actual hard part is getting it into idiomatic, safe rust, and I don't believe the LLM port makes the full transition any easier than doing things the old fashioned way: a dual lang code base like Linux.
And compiler generated assembly code is ugly and spaghetti, unlike beautiful hand-crafted human written assembly code, the kind you see in ffmpeg codecs, ...
Doesn't matter.
It does matter, because that's the entire point of the language. You know, memory safety?
The vibe coded rust port did not get them any closer to full compiler verified memory safety. They still need to go file by file, bit by bit, and make it memory safe, at which point, why not just do that from zig.
My pushback on that would be that all of the "engineering" concering a rewrite was already done the first time around. It's a glorified translation.
I do agree that the port would've taken a lot longer without LLMs though.
And how long will it take to rewrite the Rust rewrite to idiomatic safe Rust?
Can only speak for my own project, but can give an example:
After an aquisition earlier this year I got the task of doing an SAP-Integration for the new company, last time I did this 5 years ago it was a 6 month task, but with the experience and skills ive gained since I estimated it would be a 3 month project (with or without AI, most work is just logistics, AI cant help much there).
In those 3 months I was able to not only integrate SAP but also deliver a completely modernised user-facing software for that integration. While I could have written that software myself in a vacuum it would have never been worth it financially, since it would have delayed the launch of the integration by 6+ months. Building the software post-launch of the integration would have easily taken 2.5 years at minimum.
But this is also basically a "spherical cow in a vacuum" scenario, where I was essentially acting as a solo dev, in full operational control of the project, with deep domain knowledge of the topic and an allready fully set up codebase that I knew perfectly while working down ideas I've had in my backlog for 5+ years.
> Building the software post-launch of the integration would have easily taken 2.5 years at minimum.
What you have said is correct, it lets you build software much faster. The question however is: is that software making money for the company? (Not talking about what you built but in general)
I think, with AI, companies are saying yes to a lot of things they would have said No to ik say 2020. And as a result realizing “just building it” is not the answer.
Previously your GTM team or Product team would say “If we ship some big project X, we unlock $Y in revenue” but now people are realizing that those projections were really more of a hope. So companies are spending so much more tokens and shipping so many more PRs based on hope but a lot of it just doesn’t turn into meaningful revenue, especially not in short term
Yeah, thats why I meant I was in a spherical cow-situation, this was a situation where we took over a company in the exact same field and I had 5 years of experience on which tweaks would need to be made to be able to rationalise away positions.
This sort of system only works with internal software and an unusual amount of data. If we were in the business of selling that software we could not have charged a higher price for the new version over the old, the tweak could only be unlocked because we were able to control staff hiring and staff onboarding fully to make use of the new changes.
And it quite possibly creates a huge risk for the company in the longer term, there is only one person who knows how that all works.
Honestly not super sure if that is more of a risk than it was pre-llm.
As part of the aquisition I got access to their previous codebase which was some sort of incomprehensible PHP monolith, with the persons who wrote that code long gone. Thanks to LLMs I was actually able to extract the core useful concepts (again, sufficiently deep domain knowledge that I knew exactly what to look for). Without LLMs i would have probably extracted the absolute minimum and let the rest rot.
There is no reason a dev of comparable skill and domain knowledge would not be able to do that for what I built here.
But that codebase was written by people, so you'd be well positioned to take it over. I wonder if the same would be the case if you were to take over a codebase that was written by AI.
By that point it might no longer matter though, but I suspect that such code would have a lot more exposed edge cases than one where someone actually thought things through before coding.
First, it's not three years since. The real improvements in programming ability arrived in the last 6-8 months.
Second, the Internet didn't show up much in GDP and similar measures either!
But your point stands. Where are the amazing digital products/stuff? I get that it might take time to arrive as we scale up compute and learn new paradigms. But so much infra already exists (deployment pipeliens, everyone reachable on a smartphone) that we should be seeing something.
> But your point stands. Where are the amazing digital products/stuff? I get that it might take time to arrive as we scale up compute and learn new paradigms. But so much infra already exists (deployment pipeliens, everyone reachable on a smartphone) that we should be seeing something.
I'm definitely seeing indie-sized games that appear to have had significant input from AI, though I'm not sure the balance between AI for coding and AI for assets. My experience attempting this directly suggests that the current level they work at can make very simple games as one-shots, but anything more than trivial will produce outputs only as good as the developer's combined willingness to put in effort tweaking things and taking it all one step at a time, and their taste about what "good" even is.
I'm using spare credits to build and improve an isochrone map renderer, which I otherwise wouldn't have had time for (apart from anything else, I'd have had to become skilled in JS+wasm, somewhat of a pivot from iOS). This also requires taking it all one step at a time, having UX and UI taste.
Having lived through GeoCities since before it was bought by Yahoo!, taste is… well. Most people make things that nobody else actually wants.
> Where are the amazing digital products/stuff?
I've done amounts of refactoring and fixes and written tooling that just wouldn't have happened before.
I'm not sure what amazing new stuff y'all expect but the amount of technical debt in my projects is actually going down, cause I can finally get good enough test coverage, including E2E/load tests that actually prove whether the software works and scales or doesn't - just last week I diagnosed issues with SeaweedFS failing under concurrent writes when backing Sentry and could swap it out for Garage in a day, caught by a monitoring tool I slopped together that integrates with the Sentry API, no issues since.
The environment around me has gone from drowning in tech/ops debt to sort of swimming and at least holding above water for now (cause nobody will pay for 5x more tokens).
It's also insanely good for prototyping and being able to actually explore various ideas and shoot the bad ones down quickly instead of handwaving and looking at a loaded calendar, alongside being able to address well bounded tasks in parallel, better than human developers can - like I can give 5 GitHub issues to the slop machine and have it fix all of the annoying bugs. Issue with how some data shows up? Just feed it the DB dump and let it find out what's up.
Some projects have gone from around 500 code tests to around 4000, and before anyone says they're meaningless, at least 5% of those have caught real issues and helped a bunch, alongside linters and other tooling (including some tools I wrote myself). I've also written both native utilities and some web platforms for myself, side projects that I never would have gotten around to.
I'm measurably more productive than I've ever been (since I did measure that, looking at my commits over the last 2 years) but also burnt out. Still, it's the kind of burnout that's the consequence of context switching and lots of work, rather than the kind that I had years ago, where I had to manually untangle deeply nested Spring Boot service logic all over the place at like 2 AM cause the made up deadlines were kicking my butt.
In contrast to others, I don't need to move the goalposts - the productivity for me is here and now. Any future models will just make it better, unless we experience model collapse.
Disclaimer: you do need a LOT of code tests and validations, otherwise it all goes to shit. Maybe I'm just extending how much time it will be until it goes to shit for me as well, but go figure. You also have to babysit the models more than anyone would like or should, most of my work usually has 20-60 minutes of planning before dispatching the agent.
For that you need a management that actually lets you use the token budget for stuff like refactoring, fixes, and test coverage. Unfortunately I'd wager that's an exception and the rule is (still) that to most managers LLMs are "feature machine go BRRRRT", leaving even less time for anything else besides pushing out features at a breakneck speed because you're supposed to be 10x more efficient now.
I remember people moving the goalposts like that since, let me check... when did AI Dungeon go viral? Wow, 2019.
What's moving the goalposts? I am very much amazed at what Fable can do. I push its code straight to prod.
But I am just as amazed with how little real life consequence it seems to have! Even software houses were hit more by interest rates than by this magical revolution.
If I couldn't directly observe Fable in action, I wouldn't believe in AI.
I read the lack of real-life consequences as a sign that gains through digital technology as a whole become ever more marginal.
What would produce huge "gains" (depends on who you ask) again is the automation of the myriad non-digital roles/companies, but good luck with that...
> What's moving the goalposts? I am very much amazed at what Fable can do. I push its code straight to prod.
>> What's moving the goalposts? I am very much amazed at what Opus 4.8 can do. I push its code straight to prod.
>>> What's moving the goalposts? I am very much amazed at what Opus 4.6 can do. I push its code straight to prod.
>>>> What's moving the goalposts? I am very much amazed at what GPT5 can do. I push its code straight to prod.
>>>>>> What's moving the goalposts? I am very much amazed at what Opus 3.5 can do. I push its code straight to prod.
I am definitely not impressed by Opus. Quite the opposite usually.
Definitely something I've come around to as well, even though I am very pro-AI and think it's an amazing technology. But the bottlenecks just hit too hard. I notice every day where my usage of AI is so constrained, by myself and my imagination, by external factors like slow feedback loops (gathering requirements), by the AI still not being good enough for certain details (you need to heavily steer it, give feedback, even in long running agentic sessions).
So yes, many aspects of my job are now 10x as productive, but turns out that improves my overall throughput only very little.
I think it's more that it shifts the thought process from "If this is going to take 30 years then we won't bother, because the investment can be spent on things that pay off sooner" to "If we can do this in 3 years then we'll make that investment because that's a good bet."
The way it changes the game is by lowering the cost of making radical bets so we end up trying more moonshots.
Do you have examples of such moonshots? Ideally not themselves AI related
Right now we are in the golden age where we do the same and take time off. Employers have not yet fully caught up with the workforce. I can't think of people that are not putting less hours this year for the same salaries.
The questions is what happens when they catch up. They'll cut like 50%+ of the workforce? What happens then to the demand that makes their companies work?
Or an example of MS - their main cost like most software companies are people, especially software devs, which are to be replaced by AI so on the surface they would greatly benefit from it. But their products are centered around helping out people do stuff on the computer. Why would you need that when the AI will do it better and faster directly operating on the data or using e.g. Python?
Who knows. What I tell is that the AI productivity boom is here. Just for a change right now it is not shown on businesses balance sheets, because it is captured by the workforce in non monetary ways.
> where's the big deliverable from AI ... AI is simply "very useful" rather than being a historical game changer for humanity.
The last 30 years of tech has been about big, the dot com boom, the next unicorn, the magnificent 7 or 11 or whatever.
The thing is that the missing solutions people want have never been the one sized fits all multi Tennant we need 6 rounds of seed funding kinds of problems.
The "app boom" should have been a clue that smaller, niche services are viable.
IF tech were like construction, then most of HN, most of SF Bay Area, most of "tech" seems to be concerned with building the tallest building, or a massive housing development. Your average office worker is more akin to a home owner, who has a busted outlet, a sticky door, a dripping faucet, they just work around it because the resources to fix it (programers) are going to build the next big thing.
Those workers now have tools to address those problems, themselves.
I have lawyers and writers (far removed from tech) talking to me about cron jobs and scripts they are running that solve real problems for them. AI has allowed them to accomplish this.
These things arent showing up on GitHub. They dont make quarterly reports at a publicly traded company. The mountain(s) of change are happening by stacking grains of sand, not one solid rock.
> If AI makes us 10x more productive, the 3 years since its popularization were enough for a product that would have taken 30 years to build without AI, for instance.
A product still requires a lot of handholding and human thinking, at least if one does not want everyone even throwing a glance at it to immediately be repulsed by the usual AI slop tells.
> I'm an AI advocate but that question makes me feel that AI is simply "very useful" rather than being a historical game changer for humanity.
It absolutely already is a historical game changer on par with the Industrial Revolution when it comes to the amount of jobs destroyed and economies screwed up - and the impact will be even worse in 10+ years as existing seniors retire but no new seniors rise as AI has destroyed entry level career paths.
> It absolutely already is a historical game changer on par with the Industrial Revolution when it comes to the amount of jobs destroyed and economies screwed up
Which economies are already screwed up?
IMO it can’t ever be on par with the Industrial Revolution because AI can only really affect the information economy. Things people do with their hands/bodies have either already been automated or can’t be with current tech. If you’d asked people decades ago they might say no one will ever work in factories by 2026 because they’ll all be automated. It didn’t work out that way. I think AI will go the same way: absolutely game changing to some industries (of which software engineering will be one) but a great many will still survive with less dramatic changes.
If anything it might result in more focus on the human aspects. How many people out there earn their stripes putting together slide decks? In a world where an AI can put together the snazziest presentation you’ve ever seen in a heartbeat it’s going to matter more how you stand at the front of the room and present those slides than it does today.
Not disagreeing with you, but adding to my argument. Despite the human handholding, I feel that 3 years would have been enough for 18 months of thinking about the product plus 18 months where a team could get 3-5 years worth of coding/development done. Yet, we haven't seen anything big yet, like a new Youtube/Instragram, an amazing videogame, a major cure etc. Maybe they are coming, but every day that passes is an indication that the net productivity positive of the technology isn't as big as advertised. In my own personal use of AI, I experienced a 30-50% increase in productivity, but no more than that.
It's only been at most the past year where AI has been unambiguously helpful and not a hindrance. 3 years ago it gave the appearance of being helpful but it tended to be more of a hindrance.
perhaps being 30-50% ‘busier’ is something quite different from being 30-50% more ‘productive’ and ppl are just bad at detecting the difference
Maybe so, but least for me in my personal coding projects, I'm going through my own task list noticeably faster. I had created this list before getting a ChatGPT Pro subscription and the number of bugs found in AI reviews and rate of closing tasks has certainly increased. I'm not saying my anecdote scales to teams or even other people, but I have no doubt about my personal productivity change. I wouldn't be paying for it otherwise.
IMO the constraint is that AI is still rather anaemic at sustainable green-field projects: It's good at one-off oneshots, and also at refactoring or fixing bugs or adding features to existing projects, where test suites and significant architectural scaffolding already exists, but the more you move away from that, the more wobbly the results get, and the more the human once again becomes the bottleneck, for all the hard work of coming up with all the conceptual scaffolding in the first place. Typing speed rarely was the bottleneck there anyway.