At this speed this is only suitable for time insensitive applications..
I’d argue that chat is a time-sensitive application, and 13 tokens/s is significantly faster than I can read.
I mean it's a raspberry pi...
I’d argue that chat is a time-sensitive application, and 13 tokens/s is significantly faster than I can read.
I mean it's a raspberry pi...