> Non-trivial coding tasks
A coding agent just beat every human in the AtCoder Heuristic optimization contest. It also beat the solution that the production team for the contest put together. https://sakana.ai/ahc058/
It's not enterprise-grade software, but it's not a CRUD app with thousands of examples in github, either.
> AtCoder Heuristic optimization contest
Optimization space that has been automated before LLMs. Big surprise, machines are still better at this.
This feels a bit like comparing programming teams to automated fuzzing.
In fact not too rarely developing algorithms involved some kind of automated algorithm testing where the algorithm is permuted in an automatic manner.
It's also a bit like how OCR and a couple of other fields (protein folding) are better to be done in an automated manner.
The fact that now this is done by an LLM, another machine isn't exactly surprising. Nobody claims that computers aren't good at these kinds of tasks.
> It's not enterprise-grade software, but it's not a CRUD app with thousands of examples in github, either.
Optimization is a very simple problem though.
Maintaining a random CRUD app from some startup is harder work.
The argument was about “non-trivial”. Are you calling this work trivial or not?
> Optimization is a very simple problem though.
C'mon, there's post every other week that optimization never happens anymore because it's too hard. If AI can take all the crap code humans are writing and make it better, that sounds like a huge win.
Simple is the opposite of complex; the opposite of hard is easy. They are orthogonal. Chess is simple and hard. Go is simpler and harder than chess.
Program optimization problems are less simple than both, but still simpler than free-form CRUD apps with fuzzy, open ended acceptance criteria. It would stand to reason an autonomous agent would do well at mathematically challenging problems with bounded search space and automatically testable and quantifiable output.
(Not GP but I assume that's what they were getting at)
> If AI can take all the crap code humans are writing and make it better, that sounds like a huge win.
This sort of misunderstanding of achievements is what keeps driving the AI mania. The AI generated an algorithm for optimizing a well-defined, bounded mathematical problem that marginally beat the human-written algorithms.
This AI can't do what you're hyping it up to do because software optimization is a different kind of optimization problem - it's complex, underspecified, and it doesn't have general algorithmic solutions.
LLM may play a significant role in optimizing software some day but it's not going to have much in common with optimization in a mathematical sense so this achievement doesn't get us any closer to that goal.
Compilers beat most coders before LLM were even popular
had to scroll far to find the problem description
> AHC058, held on December 14, 2025, was conducted over a 4-hour competition window. The problem involved a setting where participants could produce machines with hierarchical relationships, such as multiple types of “apple-producing machines” and “machines that build those machines.” The objective was to construct an efficient production planning algorithm by determining which types and hierarchies of machines to upgrade and in what specific order.
... so not a CRUD app but it beat humans at Cookie Clicker? :-)