I wouldn’t exactly put it like that. It’s moreso the model sometimes outputting non-optimal tokens in a way that’s detectable if you know the algorithm.
It seems possible for that to make the response “drift” far from what it would’ve been, because it’s constant entropy that adds up after time.
(However, according to Anthropic and Google, it doesn’t really impact the quality of responses. I find that a bit hard to believe, although those guys are much smarter than I.)
> according to Anthropic and Google, it doesn’t really impact the quality of responses. I find that a bit hard to believe, although those guys are much smarter than I.
I would guess "it doesn't impact the quality of responses" was guaranteed to be claimed before they even implemented any of the watermarking.
And would come from marketing, not the people who implemented it.
What next token is “optimal” is fuzzy and subjective. All transformer based models have a “temperature” setting whose sole purpose is to randomly make choices other than the most likely next token. This is crucial to good output, but you wouldn’t call those choices “non-optimal” even if they are less likely. In any text generation task there are constant opportunities to make a choice from equivalent options.
Yeah, it’s hard to believe, specially when you are coding and there’s only one best way to do things, unless it plays with variable naming, or comments
Well the algorithm only increases the chances of certain words being chosen/not chosen, rather than guaranteeing it. If there’s a clear answer then that nudge won’t do anything.
If the model’s most recent output is “for (let i = 0; ”, the likelihood of the next token being “i” is probably millions of times greater than any other possible token. Thus even if “i” is on the red list and has its likelihood decreased, it’s not going to suddenly choose another word.
Put another way, on low-entropy tasks like coding, this style of fingerprinting is less effective and needs bigger sample sizes to be recognizable.
That said, even small changes can dramatically affect output quality, which is why I’m still a skeptic.
One way to watermark (assuming temperature is otherwise positive) would be to output the most likely (or optimal) token every so often.
But wouldn't you have to know the exact context before this token in order to verify the watermark? I.e. a paragraph wouldn't be sufficient; you would need the system prompt, previous prompts, and even hidden thinking?