The number of - “I did/invented Jev last year”, or, “here’s a version of Jev I vibed up last night” is getting a bit ridiculous.
Especially ridiculous is how the hacker news crowd seems to be taking these at face value…
There was one the other day with a compelling demo. But when you looked closely at it, it was feeding in the options with the word “best” on the option to pick and a fine tuned model designed to recognise that word…
I thought about writing a blog post that used Jev as a jumping off point to talk about calibration metrics[1], when and why calibration is important[2], and various methods for doing post hoc recalibration[3] of poorly calibrated models. I thought it would be interesting and instructive to benchmark Jev's claims about calibration, find examples where it wasn't (relative to some data set, which shouldn't be hard to find), give an example of how its poor calibration could be exploited, and then show how it could be fixed with, say, isotonic regression. That would give practitioners a roadmap to using Jev successfully without blindly trusting it.
But I'm probably never going to write that article because, as your comment correctly points out, the tenor of the discourse isn't very healthy. Everyone is either "clowning on" Jev (as the kids say), or gulping down industrial quantities of Kool-Aid, or talking about the hype and branding instead of the math. I'll have to find a less contentious example if I want talk about calibration.
Edit: I just found something interesting: there already is a preprint case study[4] along the same lines as the one I outlined, posted just 2 days ago!
Unsurprisingly, they found that recalibration helps enormously, as you'd expect. I guess I could still talk about the Dutch book stuff if I gave a gambling/investing example, but that's pretty well-trodden territory. So now I have even less interest in writing that blog post.
[1]: https://en.wikipedia.org/wiki/Brier_score#Decompositions
[2]: https://en.wikipedia.org/wiki/Dutch_book_arguments
[3]: https://scikit-learn.org/stable/modules/calibration.html
[4]: https://arxiv.org/abs/2609.24052
No I'm s̶p̶a̶r̶t̶a̶c̶u̶s̶ Jev!