This is classic goalpost movement. Arc-AGI-3 was launched this year with roughly 0.5% success for frontier models. being able to 99% it in less than six months sets a new record for Arc-AGI saturation timeline. Speaking of singularity measures. It is definitely a big deal, not least in that Chollet needs to cancel his summer vacation and write Arc-AGI-4 now.
Arc AGI are simple games, the hardness comes from the input being basically adversarial to LLM training. if you use an LLM scaffold that removes the adversarial part you are measuring something else.
the harness basically outsources the alien nature of what the LLM is asked to do to algorithms it writes. this would actually be impressive if you got it to do that for a much more complicated game than Arc.
with this harness the ARC AGI test becomes a test of whether or not the model can work out the transition rules in a very simple game.
A great deal of mathematics is transforming nonlinear problems into linear ones and solving them with linear techniques. Others are solving non linear problems through stochastic methods. In almost all cases most non trivial math is done by transforming a harder problem into a simpler one.
I get what you mean in terms of testing the model itself to see its improvement in some domain. However if you can transform the domain to be better adapted to the model and achieve the desired results, this is indeed an accomplishment because a whole domain of problems is shown to be practically feasible with this technique without expensive model improvements. Of course the benchmark still exists without the harness, but the harness also exists which allows these problems to be solved.
As noted elsewhere the models themselves were used to build the harness, which means the models can in fact score this scores without intervention but building a harness for themselves adapted to the domain and using it. Is this cheating by the goal posts you’re setting?
There’s a real tension between “I want to solve problems and this technique shows how to solve the problem domain,” and the “I want to measure how something performs unassisted with other techniques.” Fortunately it’s not a mutually exclusive situation. You can do both simultaneously, gain the benefit of the technique to transform the problem into something tractable and keep measuring using the benchmark.
To quote the people who make it:
> ARC-AGI-3 is an interactive reasoning benchmark which challenges AI agents to explore novel environments, acquire goals on the fly, build adaptable world models, and learn continuously.
This harness does nothing to actually accomplish those goals.
It's a clever trick, sure, but you aren't allowed to use a calculator on your basic algebra tests in school for a reason.
I don't think we got continuous learning here, but we very specifically got interim goal setting and custom world models; the thinking traces demonstrate this round trip of building a world model, mental or coded, then stopping when reality doesn't correlate, then hypothesizing and creating a new model.
The point of Arc-AGI-3 is to measure model performance. We already know that models can one-shot and iterate on very rudimentary game implementations. And, naturally, once it effectively has a copy of the source code, it can use that to play the game better.
This harness is really moving the goalpost by defeating the entire point of the test. Instead of seeing the strength of a model's world view, its ability to internally derive and intuit rules, and its ability to keep track of game state over time, we're just letting the AI cheat. This is just the LLM equivalent of running a chess engine to the side.
And this harness would not work in a remotely complex game and relies on the fact that Arc-AGI-3 is a focused test that only made the games as complicated as they needed to be for current model performance.
I think this is just too simplistic a take; Arc-AGI-1 was wide open to all models, harnesses, etc, and had quite a lot of innovative structures implemented by hobbyists. At the time, this was seen as a good thing (it was), because we don't know the best system architecture for all sorts of problems right now -> innovation is good.
The games are designed to allow assessment of a system. Knowing better systems to solve the games is a step forward. If any of the frontier labs could have one-shotted -3 in March with a custom harness, they would have done so.
Sounds like a distinction between sport and work. How useful is pure model performance if it's known that there are conditions in which even greater performance can be achieved on real tasks? How useful is it to know how fast/far a person can run if they can ride a bicycle or drive a vehicle?