So, its not a general language model, focused on tool call strictly for tiny edge-devices. There are solutions everywhere for high-capacity devices, Needle is for sub-$200 devices.
So, its not a general language model, focused on tool call strictly for tiny edge-devices. There are solutions everywhere for high-capacity devices, Needle is for sub-$200 devices.
It seems to me that the model struggles to have enough general intelligence, knowledge, or reasoning capacity for arbitrary prompted tool calling. At this size, not surprising.
I am VERY interested in seeing how it could perform with some fine-tuning for a specific family of tools/tasks. That would be a great addition to the demo.
It also seems to have far more tokens per second than needed for general "close the blinds" "tool_call(blinds, CLOSED)".
I do wonder if more smartness could be had by using sparser experts.... And possibly even having some kind of expert switching penalty to try to reduce the amount of data read from read only flash memory by encouraging subsequent tokens to use already loaded experts.
i would assume a model this size would require finetuning tbh. even functiongemma recommends that.
[flagged]
14mb? More like sub-$20 devices.
Most pi pico's come with 16mb of flash. I wonder what kind of performance that can eek out.
Well running from QSPI flash (even the internal memory versions use SPI internally) so any inference would be very slow streaming from that compared to RAM. The featured article says: “With a peak session RAM around 28MB, Needle runs on newer microcontrollers like ESP32-S3.” So I don’t see this doing anything useful on a Pico. The Pico 2 (RP2350) for example has 520k of RAM.
An ESP32 has the same amount of SRAM as the Pi Pico. You can hook up PSRAM to the Pi Pico just like ESP32 to get 16MB more RAM.
[dead]