> Claude is obviously more expensive, but it caught entirely hallucinated sentences created by Mistral OCR 4.0, so I was glad for the backup check.
What does this entail? What does Claude do to decide that the text it was provided was hallucinated? Are you telling Claude that the source was OCR'd by another LLM?
I'm basically doing the OCR twice, except in the Claude proofreading pass, it is not being asked to transcribe the document to a Markdown file. I'm pointing it to the same image input files, and to the Mistral OCR transcript Markdown file (it knows it's a Mistral OCR output), and ask Claude to check that the text is correct and point out the errors - and then make the necessary edits.
I can't speak for Mistral OCR 4.1, but the hallucinations in 4.0 were so egregious (just completely making up new sentences in the middle of a page) that I knew I can't trust Mistral OCR on its own.
How bad is the first OCR pass allowed to be to still count as proofreading? Can you let Claude compare the images with /dev/random and make the necessary edits to correct differences?
Hmm, that's an interesting idea. But it's the classifier that is triggering, and it triggers specifically on Claude's output. So I think the /dev/random case wouldn't work, because that gets Claude into the state of just reproducing the entire text from the original again.
It doesn't always get flagged. Single pages are almost always okay. Running a program that sequentially runs single pages through the API is often not okay - I wrote a program in the early 4.x days before the rule came in, that's how I hit it first. But I've also had entire articles go through just fine recently in a Claude Code session (I'd forgotten about Anthropic's rules!), and then others where I get classifier errors by page 4.
The Mistral OCR errors were small in size. Single sentences, formatting errors, paragraphs with newlines. So this was a genuine proofreading job with small changes. For the most part Mistral is actually good, but I can't have it just inventing sentences in the middle of a document. That's where the Claude proofreading pass was most helpful.
I assume they pass the image alongside the text to redo/recheck the work. Yes this is silly, but is apparently required to get around refusals.
[flagged]
Yes, that's how I read it as well, but the question remains in how did Claude know this? It was not clear to me that Claude was being presented the original scan. The entire exercise is farcical, but it wasn't as clear exactly how farcical it was that the person was essentially running OCR wastefully on a different model just to convince Claude to do the OCR as desired because of Anthropic's rules. Here's this cool tool that is supposed to make you do less work and be able to do it faster and be able to do it more accurately, except you first have to do all of this other unnecessary work with a totally different company first. <face-palm>