This is cool. I definitely think the "micro" sized LLM space is underappreciated, so it's always good to see work like this. I foresee a paradigm in some contexts where you have a hierarchy of LLMs, with more competent models actively training smaller models to solve specific tasks very efficiently, and something like this could be the smallest layer in that stack.
With that being said, the web demo is not particularly impressive. It really doesn't like anything I throw at it. I'm fine with accepting that fine-tuning is the solution to this, but I wonder if there's anything to gain from a bigger model? I know it's completely counter to the whole point of this, but a 14MB binary using 28MB of RAM seems unnecessarily small and pretty arbitrary.
Like, what does a 28MB binary get you? Or a 140MB binary? Or a 1.4MB binary? I'm guessing the choice of 14MB came from minimizing the size as much as possible while meeting certain requirements/performance expectations, but even a Pi 5 has plenty more room to spare. Curious if there's a good explanation for this (which I may have missed in my skim of the post).
So, its not a general language model, focused on tool call strictly for tiny edge-devices. There are solutions everywhere for high-capacity devices, Needle is for sub-$200 devices.
It seems to me that the model struggles to have enough general intelligence, knowledge, or reasoning capacity for arbitrary prompted tool calling. At this size, not surprising.
I am VERY interested in seeing how it could perform with some fine-tuning for a specific family of tools/tasks. That would be a great addition to the demo.
It also seems to have far more tokens per second than needed for general "close the blinds" "tool_call(blinds, CLOSED)".
I do wonder if more smartness could be had by using sparser experts.... And possibly even having some kind of expert switching penalty to try to reduce the amount of data read from read only flash memory by encouraging subsequent tokens to use already loaded experts.
i would assume a model this size would require finetuning tbh. even functiongemma recommends that.
[flagged]
14mb? More like sub-$20 devices.
Most pi pico's come with 16mb of flash. I wonder what kind of performance that can eek out.
Well running from QSPI flash (even the internal memory versions use SPI internally) so any inference would be very slow streaming from that compared to RAM. The featured article says: “With a peak session RAM around 28MB, Needle runs on newer microcontrollers like ESP32-S3.” So I don’t see this doing anything useful on a Pico. The Pico 2 (RP2350) for example has 520k of RAM.
An ESP32 has the same amount of SRAM as the Pi Pico. You can hook up PSRAM to the Pi Pico just like ESP32 to get 16MB more RAM.
[dead]
> Like, what does a 28MB binary get you?
For one thing, on beefy-enough recent CPUs, you could keep the weights hot in the L2 cache of a single CPU core. (Which is clearly not the use-case, but might be interesting to those looking for extreme TPS numbers. Or perhaps for efficient training!)
I'm quite impressed by the results of the web demo, especially given its size and the precision with which it uses the three available tools (tested with German commands). I could imagine that this LLM would fit well into a setup with multiple micro-sized LLMs for different purposes; so 14 MB for precise tool invocation is a reasonable memory footprint when a number of other local models are running (e.g. STT, TTS + language models).
yes, that's what we had in mind while building
It's called a LM :). LLM stands for Large Language Model.
LLM as in Little Language Model
Ok we'd be adopting Little Language Model officially haha
TLM
lLM