I had an issue. A documents folder with over 12k objects in it. A hodgepodge of folders and sub-folders. That over time had created a mess that no amount of file movement was ever going to make it usable. I wanted: 1) To keep my data local 2) be able to filter out PII and other data 3) Be able to find and delete duplicates 4) Get short synopsis of what a document is 5) Semantic and keyword search 6) All of this kept local to me requiring no internet access and no tokens spent to train someone elses AI.
The result I call DocuBrowser and in it's current form is FOSS (GPL-3) licensed for your personal use. The UI is in your browser. The AI models used are held local and are tiny, Available for Linux(RPM,Deb, and tgz) Windows and Mac. Let me know what you think and thanks for taking the time to try it out.
Sounds similar to https://docs.paperless-ngx.com/
Key difference I see is that you point it to a folder instead of uploading to a system.
I think paperless devs are working on AI integration, and there are 3rd party solutions. I'm holding out for an official one, so far.
It's pretty cool, I've set up a share where the scanner scans, and it automatically picks it up from there and ingests it into the system.
This looks really cool. Can you tell me the minimum specs required to run this? It would nice if you could add it to the readme as well.
I've run it on a VM with 4G ram and no GPU. It runs, But I really recommend 8G ram at least. If you have a GPU (like I do) with 4G vRAM that is ideal. Will get this in the readme. Thanks for the suggestion. I really tried to build this to minimal spec.
Could it be extended so it also extracts pictures from pptx and xlsx and run vision to get a description to be added to the text content before indexing?
Let me look into this
I've been working on something related - extracting tons of data from various formats to allow searching them - and the solution I chose for xlxs and xls files was headless LibreOffice to convert them to CSV. There's also exceljs but I found it didn't work for many old xls files.
I didn't find screenshotting of spreadsheets worked well, vision wasn't very accurate on them. I do use it for PDFs though. For docx it's probably fine either way but I went with LibreOffice -> markdown.
I went with the python libraries (pydoc and pyxls for example), because it's portable and doesn't require a big download to a users system if they don't already have it installed.
How about jpegs or other scanner images files? We have hundreds of scanned documents that were never pdf wrapped.
hmmmm :)
It’s either restricted to personal use, or it’s GPL-3. How can you have both?
By restricted for personal use I mean it's not networked. It's running on your system only. It's not a networked commercial product able to do SSO etc. It's not an enterprise level product.
Personal use? I need this at work, dragging useful info from tarpits like Teams and GitLab.
Also need to search git repos including all branches and history (TIL/xkcd#153'd GitLab's web search can basically only do one branch at a time).
But how’d you access teams when it’s work teams and don’t have api access ?
I creating DocuRepo as well. though not as fleshed out.