I've got a scan from a book that I OCR with new releases. Ligatures, critical sigla, Fraktur letterforms, subscripts, superscripts, etc.
Nothing special about this model for overly-detailed work like mine.
It's been a while since I last tested (and discontinued my subscription), but the "pro" models from OpenAI dominate. Not surprising, given the price difference, but it would be nice if an OCR-specific model could perform better. It's worth mentioning that even the highest-end models do a pretty poor job with intricate text like mine.
While I haven't tried OpenAI for OCR, I've put my small scale OCR work through both Claude and Mistral OCR. Claude is absolutely better - even in OCR work I did last week and compared with Mistral OCR 4.0.
Mistral's one advantage is that Anthropic now flags OCR, because they don't allow anything that could be considered "reproduction", even of work for which you own the copyright. So my new workflow is Mistral OCR for the actual OCR, followed by a proofreading pass by Claude (which is allowed). Claude is obviously more expensive, but it caught entirely hallucinated sentences created by Mistral OCR 4.0, so I was glad for the backup check.
Same company that OCRed millions of books, the irony.
I feel Anthropic is destroying itself with all these restriction. They got away because their models were the best for coding, but that is not an advantage anymore as OpenAI and other open source are already better.
Claude refuses to translate lyrics of songs even if it's just for your own understanding. ChatGPT allows it.
Also, Suno is so restrictive that it won't make songs of public domain 19th century poems because someone else already made a song and the lyrics are in their song database. And it won't do covers of your own uploaded humming of a novel melody because it matches "humming in an empty room" to some song fingerprint in the database.
They are all trying to avoid these lawsuits by being ridiculously overly cautious. "Your honor, look how much we went above and beyond, even when it made the product worse for legitimate use cases"
It is why they're pushing for regulatory capture. Let us do the bad stuff, make the money, then we'll regulate everybody else out of the market.
They were criticised/sued early on when people could reproduce copyright things. I dont think in this case it's something they'd prefer to do?
Yep, my understanding is that many guardrails like this are actually the result of government legislation (eg the Fable bans) or terms of settling copyright lawsuits over reproducing copyrighted text and lyrics.
I mentioned it in a sibling reply, but here's Anthropic's support document about not using Claude to reproduce content verbatim that already exists, regardless of copyright.
https://privacy.claude.com/en/articles/10023638-why-am-i-rec...
> Claude is obviously more expensive, but it caught entirely hallucinated sentences created by Mistral OCR 4.0, so I was glad for the backup check.
What does this entail? What does Claude do to decide that the text it was provided was hallucinated? Are you telling Claude that the source was OCR'd by another LLM?
I'm basically doing the OCR twice, except in the Claude proofreading pass, it is not being asked to transcribe the document to a Markdown file. I'm pointing it to the same image input files, and to the Mistral OCR transcript Markdown file (it knows it's a Mistral OCR output), and ask Claude to check that the text is correct and point out the errors - and then make the necessary edits.
I can't speak for Mistral OCR 4.1, but the hallucinations in 4.0 were so egregious (just completely making up new sentences in the middle of a page) that I knew I can't trust Mistral OCR on its own.
How bad is the first OCR pass allowed to be to still count as proofreading? Can you let Claude compare the images with /dev/random and make the necessary edits to correct differences?
Hmm, that's an interesting idea. But it's the classifier that is triggering, and it triggers specifically on Claude's output. So I think the /dev/random case wouldn't work, because that gets Claude into the state of just reproducing the entire text from the original again.
It doesn't always get flagged. Single pages are almost always okay. Running a program that sequentially runs single pages through the API is often not okay - I wrote a program in the early 4.x days before the rule came in, that's how I hit it first. But I've also had entire articles go through just fine recently in a Claude Code session (I'd forgotten about Anthropic's rules!), and then others where I get classifier errors by page 4.
The Mistral OCR errors were small in size. Single sentences, formatting errors, paragraphs with newlines. So this was a genuine proofreading job with small changes. For the most part Mistral is actually good, but I can't have it just inventing sentences in the middle of a document. That's where the Claude proofreading pass was most helpful.
I assume they pass the image alongside the text to redo/recheck the work. Yes this is silly, but is apparently required to get around refusals.
[flagged]
Yes, that's how I read it as well, but the question remains in how did Claude know this? It was not clear to me that Claude was being presented the original scan. The entire exercise is farcical, but it wasn't as clear exactly how farcical it was that the person was essentially running OCR wastefully on a different model just to convince Claude to do the OCR as desired because of Anthropic's rules. Here's this cool tool that is supposed to make you do less work and be able to do it faster and be able to do it more accurately, except you first have to do all of this other unnecessary work with a totally different company first. <face-palm>
>Anthropic now flags OCR
I haven't seen any difference in my ocr workflows, what do you mean by this?
Not the person you’re responding to, but I’ve had Claude refuse to OCR pages from in-copyright books. I was sometimes (but not always) able to get around that by changing models, by telling it that I was doing the text conversion only for personal use, or by first telling it to use Tesseract or another OCR engine to do the initial pass and then having a Claude subagent proofread and clean up the OCR output.
I’ve also had it refuse to OCR public-domain books that included content that it didn’t like, such as references to prostitution in 19th-century books about Japan.
I had one session where Claude refused to continue after it hit some kind of guiderail restriction. I couldn’t see what the trigger was, so I started a new session, gave Claude the link to the previous session, and asked it to diagnose the problem. This new Claude said it couldn’t view the exact guardrail issue, but it did suggest a workaround that turned out to be effective.
Yeah I do something similar and Claude (well OpenAI too) just refuses to transcribe anything related to slavery and pederasty in Ancient Greece.
Are you familiar with the Thesaurus Linguae Graecae? It's a project that may meet your needs. Paywalled, unfortunately, but I assume it would be a one-time expense.
Ah, I see. Thanks for sharing. I am OCR'ing public domain .pdfs, so perhaps I haven't run into that issue. The irony couldn't be stronger though.
"it refuse to OCR public-domain books that included content that it didn’t like, such as references to prostitution in 19th-century books about Japan."
Thoughtcrime -like territory and self-sensorship. The AI safety lobby is such a vile influence on the freedom of expression and communication via technology (since AI is starting to eat up rest of technology).
I guess the main problem is positioning AI tools as "human-equivalent" creators by the big AI corps. If they were positioned simply as "better OCR and proofreading" people would attribute to them as much responsibility as they would to a - say - typewriter and we would not need to have this nonsense.
I do realize most of the valuation comes from the positioning of "our TAM is the global salary base of 50 trilion and we aim to supesede human workers in the near future" which implies they need to position this technology as "human equivalent" or that valuation is no longer as credible.
This is too funny considering they did that themselves. I’m pretty tired of these companies deciding what we can and can’t do while they act with impunity.
I also saw this when I made a (personal use only) book translation (agent learns how book 1-3 of a row is translated, then translates book 4 the same way because it is not available in my language). Claude happily translated most of the book, except a few chapters which it denied. Tried several times always the same results (with no other context about the rest of the book).
The response / conversation gets blocked by the guardrails with "API Error: 400 Output blocked by content filtering policy"
At first I thought it was something in the scanned content that was being flagged, but it was the attempt to transcribe that was itself being flagged. Anthropic mention it on their pages:
"Anthropic takes these steps because Claude’s purpose is to generate new content and ideas, not to reproduce content that already exists."
https://privacy.claude.com/en/articles/10023638-why-am-i-rec...
Side note - Claude itself is not aware of this policy, and is unable to see the API responses - the turn just ends. Which turned into a really bizarre failure state where Claude thought I was gaslighting it and kept insisting it could do the work and even had the entire text in memory. Every time it would go to show me and prove it, it would hit API Error 400. I was only able to convince Claude by showing screenshots of my Claude Code screen output so it could see that I was seeing API errors. I've never seen Claude get into that angry & snarky state before, and I hope it doesn't happen again.
[flagged]
I have been quite happy with Mistral OCR for the documents I needed to process (typeset, but old, with questionable scan quality, sometimes elaborate typesetting or, much worse, typewriter-and-handwriting approximations of it). I do not test every new model when they are released, but I did a review shortly after Mistral OCR 3 was released and it was a very good compromise: cheap, fast, and good results without further processing. I found generalist models to be way too much faf to get them to avoid unnecessary modifications to the text and report accurate bounding boxes for figures and tables.
That said, models have sometimes surprising weaknesses and a model could be terrible overall but magically work for one type of document.
I got the opposite experience very recently : tried to OCR a bunch of handwritten emails addresses with chatGPT and I had to make so many corrections that I gave up. Whereas Mistral nailed it on first pass.
Which chatgpt model?
I tested different models of the 5.6 family with different settings, although I couldn't exactly remember precisely which ones.
> the "pro" models from OpenAI dominate. Not surprising considering the price difference, but it would ne nice if an OCR-specific model could do better.
I haven't been impressed with any of Mistral's models. They obviously realized that they couldn't compete at the frontier so they decided to go for smaller focused models but even those have not been that good.
This is what I found as well.
We moved away from Cursor but I was looking for a model that would help with FIM (fill-in-middle) multiline autocompletion and people were recommending Mistral's Codestral. We gave it a shot and it was lackluster at best.. Even Google's Gemini did a significantly better job than Codestral.
Ultimately Opus-class models got good enough and I don't do much manual coding anymore.
Yet: how is pricing? Evaluating contents and routing appropriately isn't a new challenge in OCR, one of the oldest fields of applications in ML. Thus, how do the smaller open models perform in tandem with relatively pricy $/pg models & APIs? Your use case is remarkably rare relative to the volume and price sensitivity of enterprise data warehouse ops.
The big difference is traditional OCR used basic pattern matching to find text, whereas models like Mistral OCR (and GPT, etc) use computer vision instead and deep learning to parse text, math equations, and apparently in some cases extract images too.
I'd love to see some advancements in traditional OCR based on ideas and concepts we've learned from newer "OCR-like" models since traditional OCR is drastically cheaper.
I made a benchmark for handwriting recognition for a project while keeping line breaks and errors (grammar, spelling). Sonnet absolutely dominates it since a good half a year. 5.6 did not change that for me. This should also translate to better ocr.
You're the second person to mention handwriting. I think it might be handwriting-specific; perhaps Anthropic has a better corpus for this.
I really wouldn't know, though. Anthropic models barf out copyright issues for my use case, so I'm unable even to benchmark them. It's a common problem when you're scanning public domain books. Mine are reference texts often cited.
Ah okay. Ofc handwriting is not copyrighted. Makes sense, that we do not have these issues.
So is Mistral OCR the best one? Have any other OCR models caught some of what you describe? I've been kind of interested in how "OCR" type models work compared to old school OCR.
My use case isn't in the realm of old-school OCR, so it's not a good comparison, but anyway:
As another user pointed out, it's surprisingly random (task-specific). Llama Scout outperformed Gemini Flash 2.5 on a benchmark I built at the time. I didn't include an OCR models.
Mistral might indeed be the best OCR-specific model for my task, now that you ask. Funny. It's so bad at my work that I didn't register it might be the best in its category. This is just based on vibes from my single scan.
The datalab models are the best ones.
Do all the models give you the bounding boxes, block labels as this one (allegedly) do?
It's very common. PaddleOCR is enough to get extremely well-done bounding boxes, and it runs very fast on a $150.00 GPU.
There's always room for improvement, though. I suspect a tool will emerge for highly detailed OCR that implements a nested bounding-box-based multi-scale approach, effectively OCRing small sections at a time and then gradually compiling them by expanding the surface area using the bounding boxes.
I've thought a lot about implementing it anyway.
edit: I see you're asking about the block labels. Leaving the comment in case someone finds it interesting.
Whats the best open OCR at the moment?
I’ve had good experiences with MinerU and PaddleOCR-VL