This is a great example of why a PDF’s text layer is not necessarily the same as its visible content.
I’d be cautious about making “non-black means invisible” the general rule. A possible middle ground between raw text extraction and full OCR would be to render the page, map each extracted glyph’s bounding box back to the rendered pixels, and discard glyphs that have almost no contrast with their local background.
That would preserve the exact embedded text for visible characters while using rasterization only to determine visibility.
Has anyone tried this kind of hybrid approach?
I was converting some PDF documents recently and I noticed a few pages were missing their text. I wondered if the text was white-on-white. I got Claude to check it, and I was partially right. The original (image) text layers were gone, but the OCR layer remained. Lost all the font data, but most of the text was recoverable.
I agree with you, it's not safe to have a rule saying "non-black means invisible".
I should have made it clear in the blog post, but the first thing I do when processing a PDF is I identify the document_type. Then I run special rules for that document_type.
So the "non-black means invisible" rule only applies when processing this document_type.