> Ask models to search on the Internet and answer?
To me, the benefit of running small models is that they fit on your device exactly that you don't need any internet connection. It's all local and offline, so you can still consult for information in any scenario.
The real target of small models is surely the opposite: knowing with clarity how to conversationally or otherwise reason about your requirements in natural language, sound and imagery, consult APIs that interact with apps, and then have a strong, generic capability to reason about what it finds.
If you want it to know more things locally, you could hopefully distribute wikipedia or other appropriate large sources on a flash storage, properly pre-indexed, with a local tool API, and let it work its magic on that. Not least because that is as I understand it, considerably easier to incrementally update.
This is, without doubt, a very difficult target, not least because it is rather at odds with the way LLMs work on a crude level.
I thought the benefit of small models is that they are a natural language UI to whatever they are connected to.
It seems strange to me to expect a small model to answer everything correctly when every device that they are being used on support networking and knowledge is constantly evolving.
This is a very bad way of thinking of it. Small LLMs have clues about real knowledge but only surface level answers will be accurate.
True but they could query a local 50GB Wikipedia dump for factual info’s via embedding search. All still local.
I wonder if someone already made a somewhat turnkey setup like this? Small models should ideally not waste a lot of parameters on trying to memorize too much knowledge and then hallucinate / interpolate a lot of the gaps.