Crazy how a smart person like this fails to understand the gumbel softmax technique. It does not affect writing quality at all, provably. The very fact that there is generally no "best next token" with 100% certainty is precisely why the trick works (you cannot watermark a response to "respond with the To be or not to be soliloquy from the first folio Hamlet", for precisely this reason).
It seems to me like he started out mad and looked to justify it.
I'm skeptical that anybody generating LLM text is really all that concerned about optimal word choice. Or even particularly good prose. But let's pretend that person exists.
If that person tried, say, an open model and that same model with watermarking applied, I'd be eager to hear their thoughts on the prose quality. Especially if they built an experiment harness and rated a few hundred blinded examples and found a measurable difference.
But getting this upset in advance of any demonstrated problem? It really seems to me like the point isn't the point
> It seems to me like he started out mad and looked to justify it.
that's been his thing since it was just a blog about apple product speculation and update. It's always been tedious.
Google has A/B tested watermarking on millions of responses. They say they observed no difference in user behavior.
It says they observed no difference in people clicking thumbs up or down. There are loads of other behavior that they didn't observe; like, say, switching to a different LLM.
How exactly do you propose they should keep track of quality, then, if not by A/B testing?
The question here is not to what product management advice to the Gemini team. The discussion here is whether the watermarking is noticeable.
The easy thing to do here would be to have 1000 questions, randomly assigning one half to an LLM with a watermark, and the other half without. Then show people pairs and say, "Which one seems watermarked?" (Or, "Which text seems more natural" or "Which is a better answer" or something like that.) If they come out equal, the watermark really is indiscernible, at least to most people.
> It seems to me like he started out mad and looked to justify it.
Yes, but that's neither surprising nor a reason to dismiss the anger. People get angry about DRM schemes in video games, even if the slowdown these cause is practically imperceptible. They're angry -- and Gruber acknowledges that factor too -- because a stranger manipulates what they regard as their own domain, without consent by or benefit to the owner.
It might be another instance of consequentialism vs. honor ethics. Many consequentialists don't seem to understand that something that doesn't have demonstrable consequences can still have moral implications.
> People get angry about DRM schemes
Yes but thats a thing that degrades something in a catastrophic way, as in I can use the thing one day, and not the next.
A different randomisation system on something that is a text generator which is designed to be unperceptable sounds like the people who are annoyed at FLAC vs MP3[1]
Done right you won't know the difference, done badly and you will.
[1] ex audio engineer, try me.
> the people who are annoyed at FLAC vs MP3
What’s the story there? I didn’t know that was a thing and I’m curious to learn more.
MP3 is lossy. FLAC is lossless. So obviously a certain type of people are going to make a religious war out of it.
> try me
Good luck explaining that one
> People get angry about DRM schemes in video games, even if the slowdown these cause is practically imperceptible.
Any potential "slowdown" doesn't even come close to making the list of top reasons people get upset about DRM.
Even if honor were real, LLMs do not have honor, and people using LLMs to write without disclosing that fact (or indeed at all, to some purists) do not have honor either.
> because a stranger manipulates what they regard as their own domain, without consent by or benefit to the owner.
It's LLM output! It's not your domain, it's the LLM owner's!
I assume they are just passing off AI prose as their own and don't want anyone to be able to tell. Which is surprising for someone who's been blogging for a thousand years. But I don't really see any other reason for this amount of heat and FUD.
Considering Gruber's always comically butthurt about regulation, especially EU regulation, your theory seems accurate.
>fails to understand the gumbel softmax technique
I think in this case it doesn't help that there are multiple watermarking schemes, and the easiest for people to understand is the red/green scheme by Kirchenbauer et al. (https://arxiv.org/pdf/2301.10226), which does technically distort the logits (but I'd argue only in cases where you wouldn't notice it anyway).
I wasn't aware of this gumbel softmax scheme, it seems you're referring to https://simons.berkeley.edu/talks/scott-aaronson-ut-austin-o... ? That's really clever as it doesn't even distort the logits, basically cryptographically indistinguishable from a "real" random sample unless you have the key.
The actual scheme Claude uses seems to be neither of those two though, they say it is SynthId-text which seems to be tournament sampling based.
By definition watermarking narrows and biases the response distribution. Clever algorithms might reduce the perceptual impact and minimize some cherry picked metrics, but it's still worse.
>It does not affect writing quality at all, provably
Prove it, then? It's not a claim that GumbelSoft paper makes: "Regarding generation quality (perplexity), GumbelSoft shows relatively low perplexity"
https://arxiv.org/html/2402.12948v3
If this is true then the probability of the detection tools flagging completely human generated text as AI generated is non-trivial. Let's say I write a completely original piece and the detection tool says there is a 36% probability it was generated with Claude. What then? Now it's up to the person looking at the score to cast a subjective judgement. Maybe to me, anything over 25% is unacceptable. Maybe to someone else, it must cross over the 50% threshold. This is the problem.
Cognitive surrender.
> the probability of the detection tools flagging completely human generated text as AI generated is non-trivial
How does that follow? AI-generated text is already not a perfect emulation of human writing. There's lots of room to affect it laterally without changing the level of quality.
As I understand it, LLMs with temperature >0 can select from many possible outputs. All they're doing is limiting the possible outputs to ones that contain this pattern. I don't see any reason why the quality of that subset should be lower than average. The very best outputs will likely be eliminated, but so will the very worst.
To sample from the probability distribution you already need random numbers.
If you get your random numbers from a cryptographic PRNG, then to notice the difference between that and 'real' random numbers even in theory, means you need to break the cryptography. In practice, your gut feeling about how good some text is won't break modern cryptography.
That's an interesting problem to discuss, but unfortunately TFA spends no time discussing that.
Yeah. And the detector can’t even tell you “X% chance this is watermarked” because it doesn’t know the input distribution. It can only tell you “Y% chance that an unwatermakred text would score this high” and how many history professors understand Bayes rule well enough to understand the distinction?
Worse, what will academic institutions decide is the threshold for detecting AI generated work. If you have a false positive how do you prove it was a false positive or we all just trust the watermark detector over the student saying "I swear I did it all by my self"
I don't think that's true. I think it's a binary 0% or near 100% probability of a watermark having been detected; the more changes to the text having been made after the text was output by the LLM and the less leeway the LLM had for probable word choices, the longer the passage necessary to see it.
The "problem" is that seeing the watermark doesn't mean that the person claiming to be the author didn't make extensive changes to the output of the LLM, or that the LLM wasn't simply the final editor of something that the author had put a lot of work into.
> Cognitive surrender.
I don't know what this means. It's just drama. Don't let the LLM write for you and this is not a worry. I'm not worried about the poetry of LLM output being subtly adulterated.
No, watermark detection is not binary, you get a real number. You decide on a threshold when looking for the watermark. This is the problem - by random chance, some human text will be detected as watermarked. You can turn the detection threshold up until it guarantees <0.001 false positive rate at the expense of higher false negatives, but seems inevitable that someone gets wrongly flagged.
Making blog posts about AI that make it apparent that the tech is going whoosh is a choice.
> The very fact that there is generally no "best next token" with 100% certainty
Indeed.
It's frankly bizarre to see the assumption to the contrary being made by someone who's been passionately blogging by hand for years, who also happens to be responsible for the notoriously vague, humanistic, DWIMmy Markdown standard.
"Smart"? Have you read his writings in the last decade? It's all nonsense, which is why I stopped reading circa 2018
I think he’s still generally good on business, UX, and hardware design. That’s all subjective and taste I suppose, but his taste works for me.
On deeper tech stuff, like this utterly nonsensical misunderstanding of watermarks… yeah, classic case of a guy who is smart, and has lost the ability to realize when they’re not knowledgeable in a domain.
[dead]
[dead]
I couldn't be happier that people are mad about it. To quote Calvin, "nothing helps a bad mood like spreading it around"