Kind of related to this in that I built it to hoard knowledge from web pages I've visited along with implementing a Karpathy-style LLM Wiki, but the knowledge is collected automatically from sources I browse.
I have it up on GitHub, but I don't think anyone should use my implementation.
Loosely, what I built:
* On each of my machines I have a cron job running that looks at all my web browser history (usualy it's inspecting the brower's SQLlite across firefox and chrome). If it matches my rule list: hacker news stories, certain reddits, etc. it'll grab the page, convert to markdown and drop in my Obsidian Vault incoming.
* It has a whole de-duping architecture since I might open the same page on multiple machines. Uses the CloudFlare SQLITE D1 storage for tracking the processed links.
* it'll then trigger the LLM to do some Karpathy wiki style taxonomy assignment to the articles, organize them, create an index etc.
It's then available for my "bot" stuff to do writings for me.... I will probably write more about it at some point. I'm not certain it's totally useful and not just a yak-shave on hoarding knowledge.
Ai-drafted article on this [1]
Example AI-Drafted article based on some discussions the other day on Ollma vs LLama.cpp [2]
[1] https://taude.xyz/posts/how-archivore-turns-browsing-into-a-...
[2] https://taude.xyz/posts/skip-ollama-run-llama-cpp-directly-o...
Webpages used to have meta keyword tags until spammers hijacked them for irrelevant keywords.
I think browsers can play a part in building a local search index for URLs based on those keywords the page declares and cross verify/accept only those that are in prominently visible content, or may be delegate to an external engine(like LLMs) via an extension etc. This is particularly useful for cases where full text indexing is not feasible or desirable.
I doubt Google will ever add such feature in chrome though.
Did irrelevant keywords ever work for the spammer? You could add "naked" as a keyword because it had a lot of searches compared to "compiler". Now what happened to those looking for naked cats when they got to your website about code compilation. It's not like they would stop and think "oohh I'll look at this instead"
Meta keyword tags have always sucked even without malice. Because they're invisible, authors frequently forget to update them, copying <head> boilerplate between documents.
"I can't practice minimum quality control as a web dev" doesn't sound like a problem with the standard.
When designing a standard like this you need to apply systems thinking.
The poor data quality is a problem for anyone who wants to use the tags, which has seen everyone almost universally reaching for other solutions. Search engines have preferred anchor tags, bookmarking solutions have applied user tagging.
With keyword tags it's been a vicious circle of poor data quality and neglect since day one. Even in documents from the early 1990s when people were really trying, the data quality is inconsistent at best.
I feel like this could be useful for my Karpathy-style LLM Wiki, but then again I think, in this day and age most of what I'm reading about is outdated in a few weeks. Not only are the tools/methods coming and going, but the ones that stick around drastically change. It would probably turn into a dump of antiquated markdown that could cause confusion.
As far as the need for private search, well, I've already searched for or visited those pages, so...
Yeup, as I mentioned, it kind of feels like a Yak-shave....but it was fun building it, and gave me some experience with agentic coding and new technologies I don't normally work with. It's nice to dig into for some research.
The biggest win was the realization that both firefox and chrome maintain all the links you visit in a very queryable SQLite database. I've been poking at that for a lot of custom tools, like WHAT JIRA tickets am I paying attention to this week, etc....
Why don't you think people should use your implementation? Just curious
I vibed most of it (and though I like the code that was output, I coached it with some custom skills). If I were to share it out, I'd probably clean up some of the tools to make their interfaces smaller and cleaner. And make the docs a lot simpler. I'd probably also create some form of dashboard so you can see what's happening (all the docs scraped and saved), etc.
EDIT: it's also the type of thing that feels very personally customized for my needs. I encourage you to build something similar on the idea. Much like how Karpathy Wiki was suggestive and not a runtime to just use...
I guess he meant it in a "look, I made a cool thing that is useful to me, but don't go throwing away your current organizational method just yet nor leave it open to the internet" way.
> have it up on GitHub, but I don't think anyone should use my implementation.
ditto, it's an experiment in near-vibe coding, which also uses Typesense for queries using BM-25 & RAG with fusion. I have the web search/fetch/crawl features persisting raw intermediate values (api responses, search result lists) because I might re-use them one day... at least good for auditability if I need to
related, it is using Hister author's prior project SearXNG as one of the search providers