Grep (or ripgrep at least) is i/o bottlenecked at this point. It's impossible to process data at faster than i/o speeds, since you have to get the data to the processor somehow. That doesn't change whether that processing is grep on a CPU, or LLM on an ASIC.
ripgrep may be.
I did a toy project once implementing a limited version of grep on an FPGA and was able to get some speedup over GNU grep at the time, though marginal.
In any case, LLMs aren't IO bound :))
I am now imagining a future where developers buy fancy "grep cards" for their machines. I don't hate it.
I look forward to GPGrepPU -- someone'll find a way to abuse them for scientific workloads or something.
It would be cool if they could be daisy-chained, so you can have a hardware implementation of |
The next logical step would be to have a separate card for each Unix utility.
And a patch panel and a bunch of patch cables that you plug in and out to construct your pipelines.
And eventually hire people whose job it is to patch pipelines on demand for everyone in the office.
“Hey Jim, I’m gonna output the systemd logs of nginx on line five, can you assemble a grep pipeline for me to match all HTTP 500 status codes from /api/cart POST request log lines? Connect the filtered output to Tim’s desk, line 7. He’s there now, we are trying to figure something out.”
“Sure thing Bob, give me a moment.”
I'm sure this is how some bellheads (in the phone tradition, not in the Unix tradition) envisioned computing.
It’s OK: you can just tell us you have a financially and relationally crippling Eurorack addiction. You’re amongst friends here.
Bring back IBM accounting machine plugboards!
Patching cables is so primitive... Use punched cards to define the connection patterns or something!
Transputerpunk.
Universal Basic ASICs
If your BASIC is compiled that's just a CPU!
At some point Intel was experimenting a Xeon with built-in FPGA, maybe they were just a bit too early to the game :)
Almost 30 years ago there was a project about making a computer which used FPGAs for all "software". It was called RAW.
Baring it all to software: Raw machines | IEEE Journals & Magazine | IEEE Xplore https://share.google/nI9GyFJu4HvbYFrBf
(If you Google the name you will find free PDFs as well, the IEEE page is more useful as a summary and such.)
I've evaluated some of these and in almost all cases they failed to outperform Intel's ridiculously optimized CPU regex library, hyperscan.
Right, but I read the comment's point as: being i/o bound is the eventual limit of the asymptote and it's the same one grep has.
GPUs and inference ASICS also have large amounts of high bandwidth memory, plus lots of high speed storage cache, and dedicated very high bandwidth scale-out and scale-up networks. Because they are also often bound by I/O bandwidth.
If your problem is grepping crazy amounts of data, the infrastructure for LLMs isn't a bad place to look for an example.
> plus lots of high speed storage cache
I wouldn't be surprised if that's what Apple is focused on for their next generation platforms – I wonder if more layers of caching between their SSDs and unified memory are on the cards.
i/o could still be sped up though? And I dunno, maybe this actually is an argument for how the llm version could end up being faster, because there is lots of investment in crazy fast i/o hardware and protocols to get the data into the chips.