When they say "can't hallucinate" they mean they produce a confidence value for every result, so you could see for example it has 0.1 confidence, and you can disregard the result - that'd be different from hallucinating where it believes it's correct
if it puts a high confidence value on a wrong answer, thats still hallucinating, no?
llm hallucinations are high probability tokens that are incorrect vs the real world
No, I don't believe so. Hallucinations are not "high probability" in a real sense. They are an artifact of the random walk the inference algorithm takes, which causes it to latch on to and chase attractors in the noise. This random walk behavior is necessary for chat interfaces to be useful, but are less critical to typed output predictors. I'm guessing they found some optimization that is possible if you give up caring about chat.
Correct, they have not made a universal all-knowing omniscient oracle, which is what would be required for "can't hallucinate".
That seems like a weird standard.
I would be happy enough with: only produces what it can verify with sources.
If you eg try to remember a court case (ie produce the reference via LLM token generation only), it's easy enough to check with your data whether it really exists. Similar for following links and other references.
If your data or sources are wrong, obviously your report about them will be wrong. But I wouldn't call that a hallucination.
There isn’t a single human in this world and hasn’t ever been that meets your happy-enough standard. Make of it what you will.
Why is that at all relevant?
Humans are known to hallucinate a lot. Ask 10 different witnesses at a crime scene what they saw and they'll all report different things.
A good, non-hallucinating LLM would only report things for which it has evidence. It would consult the facts every single time.
It's a pain in the butt for humans to fact-check everything but LLMs can quickly look up all kinds of stuff. That's what makes them useful.
Yes, and for the LLM you can do it in multiple passes.
So you can bolt the fact-check / source-check pass onto whatever other system you have, without having to redesign the underlying system.
It's not a binary thing. You can get closer or further away from that standard.
And humans also behave differently in different contexts. A conversation at the pub has more such hallucinations than a formal deposit in court. For the latter, a good lawyer will look at her shoes, when you ask him what colour her laces are.
Not to be tooo pedantic, but a bot that assigned 0 confidence to everything wouldn’t hallucinate.
A calculator either gets the right answer or doesn’t answer.
It wouldn’t have to be all knowing as long as it knew perfectly what it doesn’t know
A quantum calculator answers in distributions.
What we would want to see if a confidence value that is in line with the actual correctness. If the value is 0.9 for 1000 different answers, then approximately 900 of those answers should be correct.
Yes, there is no magic sauce here that makes stochastic output binary if that’s what people are looking for.
Right and so maybe we should stop saying "can't hallucinate" when it can by definition.
It’s not what people are looking for, but what they wrongly claim.
Technically speaking when you send the prefix “The capital of France is “ into an LLM it will also produce probabilities across its whole vocabulary.
The probability values don’t really represent confidence in modern LLMs though, especially after RLHF and RLVR.
System One says they use RLCD, Reinforcement Learning for Calibrated Decisions, which presumably has accurate probabilities as an explicit optimisation goal.
How is that different from RLVR?
RLVR generally upweights tokens along the whole thinking trace that led to a correct answer, whether each token was "correct" or not. RLVR doesn't train a model to output an 80% likelihood, it just trains it to produce correct answers, and not to produce incorrect ones.
System One hasn't said how RLCD works, but they do say it is explicitly training models to output "calibrated" probabilities, which makes it distinct from RLVR. This is how they describe it:
> System One models are trained for calibrated decisions: their probabilities are optimized against outcomes to reflect uncertainty.
… which they could provide in their APIs but are vehemently opposed to because it makes distillation much easier, and faster.
Yeah. Yet another reason why open-weight models are better. If I want to use the logits, I can.
that's right, but because these models are probabilistic, it's also possible to be confidently wrong (and all future models will be smarter still and still have that possibility)
I read "hallucinations" as "generates novel output with no grounding/source". i.e. "it just made something completely up".
I believe their "accuracy" metric (sonnet 5 level) is where "right/wrong" is measured.
Yeah but what stops it from producing confidently incorrect outputs...
Nothing, but imagine using LLMs for a classification task
People out there are so resigned to the models being unreliable that they are really doing things like hallucinating deliberately, and then matching the hallucinations to embeddings -
https://softwaredoug.com/blog/2026/08/10/hypothetical-classi...
You could do that or you could just... use a model that will never produce unreliable outputs in the first place.
But we're going from "Apple" to "Apple: 99% - trust me". It could still be an image of an orange :)
It's pretty darn smart. If you did want to hack on it in earnest and find out for yourself, send me an email - nathan@typesafe.ai
I'm certainly not resigned to that, at least for classification.
Even non-frontier models are absurdly good at this in a broad sense.
Which would make it hard to judge "a model that will never produce unreliable outputs in the first place" against something that is already really, really good and exceptional in domain-specific areas with the tiniest amount of elbow grease.
Speed and cost look good though (for now)!
What about the LLM calls though that are done midchain? In the Home Assistant video the multi-intent prompt gets split using what looks like a traditional llm model, which I'm assuming is vulnerable to classical hallucinations.
That's really funny when you consider that generative models also don't hallucinate if you check up on them on every token generated?