Can anyone please explain this Jev thing to me?
We’ve always had output schemas for LLMs, and we’ve had small language classifiers for decades, so what’s new? Is it just some sweet spot in between in terms of quality vs speed?
Can anyone please explain this Jev thing to me?
We’ve always had output schemas for LLMs, and we’ve had small language classifiers for decades, so what’s new? Is it just some sweet spot in between in terms of quality vs speed?
It's essentially taking output schemas as we've been using them and applying them to specific classification tasks. So not using them to generate structured content which incorporates generated text, but using them to generate structured content which includes classification and/or rankings of the requests made.
So in a lot of cases when we've used LLMs as a classification hack, we've burned a ton of tokens in reasoning and output that we didn't really need to use to interpret the final result. (And I'll just say that we may not have needed all of the output tokens, but that incorporating assessment along with scoring seems to provide more accurate results.)
This goes beyond just asking an LLM to assign an arbitrary number to a particular concept, which in most cases distributes less-than-correct statistically, although that didn't stop us from considering LLM as a judge to be a viable strategy.
So this basically gives us a different class of model to use when classification or decision making is the only need. It doesn't replace any of the narrative if you still need that. Coupled with the higher speed and lower cost, that's why everyone's excited about it.
LLMs are generalized token predictors. They generate. Jev is a generalized classifier. It does not generate. It computes probabilities, REALLY fast.
So inputs and outputs of LLMs are tokens. Inputs to Jev are state (arbitrary strings/tokens) and, depending on the type of query, either an assertion, options, or choices. (All of those are also arbitrary strings/tokens). Outputs from Jev are probabilities. If it's an assertion, the probability that it is true. For options and choices, it's probabilities for each one, basically.
Because Jev answers so quickly and inexpensively, it's a likely replacement for complex, best-effort functions like `isSpam()`, where up until now the only nondeterministic way of implementing that was an LLM, which is slow, costly, and may produce invalid/corrupt output.
As far as I understand:
1) it's very fast (they claim 40-200x faster than frontier models [1], would roughly line up with it doing diffusion)
2) each answer carries a calibrated probability (ie. frequency of outcome is close to predicted)
Another point being that it doesn't reason, hence designed for "System One" tasks.
I wonder if in continuous control with discrete actions (eg. their DOOM demo) it can make sense to blend answer by confidence instead of taking the argmax.
[1] https://typesafe.ai/blog/introducing-system-one-models-and-j...
Jev uses a different training architecture called RLCF (Reinforcement Learning from Calibrated Decisions) vs the traditional RLHF that most TF models use.
So at the end of the day the groundbreaking work wasn't the model itself inherently but the way it was trained and then the way the harness interacts with it.
So this demo here is showing the harness side of things afaict but then TypeSafe's Jev takes it a step further via a specific training regimine.
One of the biggest issues with LLMs is that they don't work well as a classifier. They tend to pick up on the patterns of the examples and not the intent of the examples (gets worse the more examples/intents).
Does Jev solve this?
who cares how it was trained.
https://x.com/MatijaSosic/status/2100190746389135772
This is a 45 second vibeslop video that tells me nothing other than “it’s a one shot classifier” which I doubt is the interesting or useful part.
Anecdotal: LLMs like the hallucinate things and did a poor job of determining when to leave things null/blank. A more structured approach with confidence ratings helps resolve.