That's not why.
It's because counterexamples are easy compared to proofs which require new mathematics.
GenAI is great at combining existing things in new ways (interpolation). It's terrible at creating new things from scratch (extrapolation).
That's not why.
It's because counterexamples are easy compared to proofs which require new mathematics.
GenAI is great at combining existing things in new ways (interpolation). It's terrible at creating new things from scratch (extrapolation).
This is the wrong way to think about mathematical (or any other kind of) creativity in my opinion. In the extremely high-dimensional space of "ideas" (whatever that means) there are almost certainly profound ideas that are the interpolation of existing knowledge, i.e. the curse of dimensionality. It's not at all clear that you need to extrapolate from existing knowledge to be creative.
>GenAI is great at combining existing things in new ways (interpolation). It's terrible at creating new things from scratch (extrapolation).
I think you believe a fallacy about how human cognition works if you think we actually do something different than interpolation
I think this is somehow related to GenAI’s issue of “confident incorrectness”. I’ve been trying to prompt GenAI with some highly challenging prompts recently, especially terse ones:
> Which Anjunabeats and Anjunadeep compilations have the most breakbeat tracks?
GenAI was able to give me breakbeat tracks that were on the Anjuna labels, but it completely flubbed on whether and which those tracks were on compilations. It was very confidently, mostly incorrect.
I think this is somehow isomorphic to the “interpolation vs extrapolation” issue. GenAI is forced to generate an answer, and there’s no mechanism by which we can interrogate the model for its confidence (at least, that’s my understanding).
And yet, I asked “How confident are you in your answer along different dimensions, ie whether those tracks are breakbeat, whether they are Anjuna tracks, or whether they are on those compilations?”
And its answer was surprisingly satisfying, it only gave 30% confidence that the tracks were on the compilations, which is where it flubbed.
Anyway, personal observations, GenAI is meant to be interacted with, not just a single prompt black box.
Out of curiosity which model & interface did you use? I'm starting to think that, for IR tasks like this, the number one differentiator among models and harnesses is the ability for the model to look at its available evidence and conclude that it doesn't know the answer. It depends a lot on how the chain of thought goes. I have seen a lot of newer models try to do this in their thinking traces, and it seems very hit or miss.
Even Opus 4.8, GPT 5.5, and Gemini 3.1 Pro (haven't tried 5.6 yet) confidently make up BS sometimes and needed to be reminded to verify thnings instead of inferring or relying on "memory" from training data.
IMO harnesses need some kind of built in "are you sure about that?" loop checkpoint that pauses the main interaction and asks the LLM to evaluate the strength of evidence for a claim it's about to make. It seems almost unreasonable to expect an LLM to next-token-predict its way to such a conclusion, reliably, without prodding. That or we just need another year of RLHF and another 250 billion parameters, IDK.
I feel like that should work becaus when you're using an LLM for a batch inference or classification type of task, it's not that hard to prompt them to emit an "unknown" flag. They just won't do it on their own, and in a coding harness there is so much else going on that a prompted reminder gets lost.
GPT 5.5
> IMO harnesses need some kind of built in "are you sure about that?" loop checkpoint that pauses the main interaction and asks the LLM to evaluate the strength of evidence for a claim it's about to make.
As a practitioner of ML for the past decade, this is the part of LLMs that has always been least satisfying for me. The shared intuition amongst myself and my colleagues has always been that the strongest models are the ones that produce actionable insights directly. Doing all sorts of data preprocessing gymnastics and then following up inference with a 1000-line wall of if-else business logic has always indicated to me that the model is somehow deficient, and not truly capturing the kernel of “truth” within the data, either because the model choice is wrong (e.g. linear regressor for nonlinear problem), or, more likely in practice, the collected data does not effectively capture the underlying “truth” of the measured phenomena with a sufficient signal-to-noise ratio.
LLMs are powerful and have essentially limitless utility, but there’s something missing at a fundamental architectural level for which harnesses are simply a bandaid. Of course, solving that fundamental architectural issue is probably the key to AGI, so…