It still works the same way other LLMs do, by outputting the probability distribution over the possible completions (The weather is ... (sunny (50%), cloudy (50%))). Then the next token is sampled from this probability distribution (in our example the next word could be "sunny" or "cloudy" equally likely), which can result in different outputs every run.
Could the model or algorithm be changed to make it deterministic somehow? It could help a lot if there were reproduceable outputs from deterministic baked-in silicon.
But since it's so fast you can just ask it 100 times where Bruce Lee was born, and statistically you'll get the correct answer. We could call it "mixture of idiots". /s
It still works the same way other LLMs do, by outputting the probability distribution over the possible completions (The weather is ... (sunny (50%), cloudy (50%))). Then the next token is sampled from this probability distribution (in our example the next word could be "sunny" or "cloudy" equally likely), which can result in different outputs every run.
Could the model or algorithm be changed to make it deterministic somehow? It could help a lot if there were reproduceable outputs from deterministic baked-in silicon.
You can make any LLM deterministic by dropping the temperature hyperparameter to zero.
This will generally make them suck, though, a little bit of randomness is necessary for proper function.
You can also use a fixed seed for your prng. A hash of the input text (up to the current turn) should do.
But since it's so fast you can just ask it 100 times where Bruce Lee was born, and statistically you'll get the correct answer. We could call it "mixture of idiots". /s