> I've tried other very small models that run on CPU only and had better results
Maybe you can you share some comparative examples?
> I've tried other very small models that run on CPU only and had better results
Maybe you can you share some comparative examples?
sure, here's my conversation with BitNet b1.58 2B4T
https://pastebin.com/ZZ1tADvp
here's the same prompt given to smollm2:135m
https://pastebin.com/SZCL5WkC
The quality of the second results are not fantastic. The data isn't public, and it repeats itself mentioning income a few times. I don't think I would use either of these models for accurate data but I was surprised at the truncated results from bitnet
Smollm2:360M returned better quality results, no repetition, but it did suggest things which didn't fit the brief exactly (public data given location only)
https://pastebin.com/PRFqnqVF
Edit:
I tried the same query on the live demo site and got much better results. Maybe something went wrong on my end?
You were using bitnet.cpp?
Yes