I built it at home this morning and tried it, perhaps my expectations were high but I wasn't terribly impressed. I asked it for a list of ten types of data I might show on a home info display panel. It gave me three. I clarified that I wanted ten, it gave me six. Every request after that just returned the same six things.
I know it's not chatGPT4 but I've tried other very small models that run on CPU only and had better results
This is a technology demo, not a model you'd want to use. Because Bitnet models are only average 1.58 bits per weight you'd expect to need the model to be much larger than your fp8/fp16 counterparts in terms of parameter count. Plus this is only a 2 billion parameter model in the first place, even fp16 2B parameter models generally perform pretty poorly.
Ok that's fair. I still think something was up with my build though, the online demo worked far better than my local build
> I've tried other very small models that run on CPU only and had better results
Maybe you can you share some comparative examples?
sure, here's my conversation with BitNet b1.58 2B4T
https://pastebin.com/ZZ1tADvp
here's the same prompt given to smollm2:135m
https://pastebin.com/SZCL5WkC
The quality of the second results are not fantastic. The data isn't public, and it repeats itself mentioning income a few times. I don't think I would use either of these models for accurate data but I was surprised at the truncated results from bitnet
Smollm2:360M returned better quality results, no repetition, but it did suggest things which didn't fit the brief exactly (public data given location only)
https://pastebin.com/PRFqnqVF
Edit:
I tried the same query on the live demo site and got much better results. Maybe something went wrong on my end?
You were using bitnet.cpp?
Yes