Text is simply not information dense enough to be able to decode some arbitrary signal of provenance from it. Sure you might be able to detect today's tells (particular sentence structures preferred by Claude, phrases, etc) to get you some arbitrary chance percentage it was machine generated, but it's a bad fiction to perpetuate that any of this is anything more than tarot card reading.
Images, absolutely, there are tell-tale artifacts from today's generators that simply aren't emitted by "natural" paths to create them, and you can "detect AI" with high confidence (for now). Words, no, the signal is far too sparse and we are well into undetectable sophistication with today's models, let alone tomorrow's.
Whether a text was written by a human or not is just a single bit of information. So you can't rule out its detectability a priori, since even the shortest text contains more information than that.
As long as LLMs are used to write texts humans wouldn't want to write if they could help it (that's why they're getting an LLM to do it, after all), they'll remain detectable. Even if the reasoning might end up equivalent to "This looks like spam; no human in their right mind would write this spam by hand if they could get an LLM to write it, therefore it's most likely written by an LLM."
That's like saying whether or not you're going to fall in love this year is just one bit of information, so you might be able to read it from astrology. Yeah, sure, it might happen for some people with a certain star sign. But across the population there is zero reason to believe that there is a) any significant correlation and b) enough data variation in to even distinguish classes of humans.
You definitely can rule out the general case a priori. If the problem were possible, for every text there would be a unique provenance label “human” or “ai”. But since humans and machines have both written many texts, it is not possible.
As an example, you could imagine a giant lookup table that deterministically mapped every text ever written to “human” or “AI”. You would very quickly run into situations where the labels conflict for the same piece of text.
The data is statistically inseparable which makes it impossible to classify from text alone.
Not all humans are in their right minds, unfortunately.
It is much harder to tell one from the other, and for oneself, than it often seems on the surface.
There are two problems, false positives and changing the LLM's pattern.
It's really easy to have a false positive and false positives can be very harmful if the person using the detector isn't aware of that risk.
It's also very easy to change the pattern of LLM output. You can provide basic prompting that will significantly change the structure of the output. For example, having it utilize the Wikipedia article on signs of AI writing and avoid everything it describes. https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing
"It's really easy to have a false positive"
Not really. The false positives for the SOTA detector are very very low.
"It's also very easy to change the pattern of LLM output."
Not in a way that can reliably avoid detection. The problem is the patterns are baked into the distribution itself. It's smoothed over, so it becomes difficult to prompt your way out of that.
Signal is easier to detect with more data to work with.
Largely AI generated books are a vastly different situation than a one paragraph homework assignment. But multiple rounds of homework assignments would change the accuracy.
> but it's a bad fiction to perpetuate that any of this is anything more than tarot card reading.
Hard disagree. LLMs (especially base ones, that only received pre-training) can produce output that is undistinguishable from human writing (because that's what they were trained to do).
But commercial chat models are specifically tuned in a way that maximizes user engagement. It's that specific tuning that is very easy to spot when reading AI slop, and that's not surprising that it's easy to spot automatically either. And I don't think that's going to change anytime soon, unless their incentives change.
(We can say exactly the same thing about man-made stuff optimized for a specific purpose, like stock photography, clickbait titles or industrial food: they aren't stereotypical because their creator lacks the skill to make them otherwise, they are like that because that's what works best).
> especially base ones
Did you actually try them? I did.They generated even more "slopey" text than instruction-tuned ones.
They're also designed to not offend anybody, so their output tends to be very bland even compared to the most milquetoast of human beings. I was only surprised once when ChatGPT responded with an enthusiastic "hell yes" seemingly organically, but 99.9% of the time these AI services clearly are instructed and trained to provide flavorless word vomit. I don't think there's a technical reason why an LLM couldn't produce totally convincing output, but internet grifters don't need to go through that trouble. It's like how most phone, email, and social media scams come off as completely transparent to most of us, but that's the whole point; we're not the target audience of the scams. Readers looking for substance, nuance, and real opinions aren't going to notice if something with written by an LLM – unless there are some cliche punctuation tells.
When DANmode bypasses were a common thing the LLMs would drift significantly far from corporate speak.
But that's the point of corporate speak, you tend not to say thing that may offend your clients and deprive the company of future revenue. Of course there are some companies that make their living being 'counter-culture' and saying what they want, but they are a small percentage of all revenue.
> But commercial chat models are specifically tuned in a way that maximizes user engagement. It's that specific tuning that is very easy to spot when reading AI slop, and that's not surprising that it's easy to spot automatically either.
There are two problems with this.
The first is that it would still misclassify human-authored text written under the same incentive, and most people have various incentives to "maximize engagement".
And the second is that then people would just make other models that are tuned for defeating that sort of classifier, which would be used whenever the classifier is being used.
All of that may be true, but pangram currently has a false positive rate of about 1 in 10000, and this has been tested by feeding in thousands of texts written before 2020.
That may not last if AI companies start trying to build models that fool it, but for the time being at least, modern models do have strong tells.
>and this has been tested by feeding in thousands of texts written before 2020.
And these text didn't train the model in the first place? I just want to ensure clarity on that.
>pangram currently has a false positive rate of about 1 in 10000
Says Panagram.
The problem with just looking at old text is language is a living thing. Say for example I make up the world 'oklambroahaha' right today. Both humans and AI pick up that word and start using it. Now lets say the model says that anything that uses oklambroahaha is 100% AI, you can't just point and say, "well my detection AI is correct on things 20 years old, so it's right skibbidy toilet 6/7".
There is a ton of evidence that use of AI changes the way we speak and write, so it will just turn these AI detectors into bullshit generating classifiers.
You can get an arbitrarily low false positive rate by sacrificing against false negatives. It's trivial to make it zero, just classify everything as human-generated. Meanwhile a false negative rate of even 1% is a pretty big problem since someone can easily use LLMs to generate 100x the volume of text and then use whichever ones make it through the classifier.
And that's before anyone even tries to get the LLM to generate a different style of text. Or for that matter creates a "style model" that rephrases text.
You don't really need a style model - current models are very good at doing "style transfer" of a model text onto whatever it has written if you just have it do it chunk by chunk. It takes more to prevent it from being detectable by good detectors, but it does remove a lot of the worst tells.
The point being that you wouldn't need the developers of the most popular models to themselves be trying to fool classifiers because their output could be run through an independent special purpose one designed to remove the tells the classifier is looking for, and the special purpose one wouldn't need to be made by anyone with the resources to create a good general-purpose model since it only has to do that one thing.
My point is that you don't need a special purpose one to achieve this.
Pangram won't know how much AI written text they fail to detect, though, and detectors is a great tool to adjust methods of generating less AI-sounding text.
> The first is that it would still misclassify human-authored text written under the same incentive, and most people have various incentives to "maximize engagement".
The thing is, humans are significantly worse at maximizing numerical goals than computers.
> And the second is that then people would just make other models that are tuned for defeating that sort of classifier, which would be used whenever the classifier is being used.
Anyone can already do that right now, just grab unsloth studio and fine-tune your local Gemma, but nobody cares. People posting slop content don't care if pangram or I flag their slop with certainty, they are using the easiest option, which is commercial chat models. And given this segment of user doesn't care, the provider have zero incentive to provide a dedicated stealth model for that purpose.
> The thing is, humans are significantly worse at maximizing numerical goals than computers.
I'm not sure this is even the right premise.
Existing LLMs try to maximize engagement, and they often write in a particular style that has tells, but these two things are not necessarily related. Over-using em-dash or whatever isn't the thing that maximizes engagement.
So the two problems really are, what happens to the actual humans whose writing style is a close match for what a given generation of LLMs output? And, what stops LLMs from using a different style when someone wants to fool the classifier?
> People posting slop content don't care if pangram or I flag their slop with certainty, they are using the easiest option, which is commercial chat models.
They don't care as long as the consequences of identifying it are immaterial, but in that case what's the point of classifying it? Whereas if they need to fool the classifier some threshold percentage of the time in order for enough of their spam to get through, they're going to care.
> Over-using em-dash or whatever isn't the thing that maximizes engagement.
It's the thing that minimizes the loss during the RLHF phase, and the RLHF phase is the one that's aimed at maximizing engagement (it's literally trained on that).
> what happens to the actual humans whose writing style is a close match for what a given generation of LLMs output?
If a human, for instance because its writing gets polluted by reading too much AI slop, matches the style of an LLM closer than a certain threshold, then his own writing is going to be flagged as well. Whether it's an actual problem or merely a theoretical one is an open question. (unlike OpenAI and Anthropic, humans writers do have an incentive to avoid being flagged as AI).
> And, what stops LLMs from using a different style when someone wants to fool the classifier?
In theory: nothing. In practice if you fine-tune your own model: nothing. In practice with commercial models: the interests of the model making company.
> And, what stops LLMs from using a different style when someone wants to fool the classifier?
Websites have pretty much stopped using ad-blocker-blockers, it seems that it's not a fight worth fighting for them. Does that mean that ad-blockers are useless?
Most people don't even care about ads, I don't think they care about slop either, that's why there's slop posts and obnoxious websites that are unreadable without an ad blocker. A slop blocker used by 10-20% of the internet users wouldn't change the calculation more than ad blockers did.
I mean, back when I was spam filtering setting up a simple Bayesian classifier was easy. Train it on your spam and ham and it worked damned good. "Mission Accomplished".... until it wasn't. Spam rates started climbing and it started getting harder than ever to filter them.
There is always an incentive to get spam to bypass filters, so as your filters increase in accuracy, those attempting to pass said filters adjust their behaviors.
Spammers/cheaters/whateverers will at least just use a second pass filter that uses one of these 'ai scoring' systems to beat said AI scoring systems. So while it's worthwhile to do it at this moment, this window will rapidly close.
I don't think it's a very good remark, as there's significantly less email spam than 20 years ago.
Another example is ad-blocker-blocker. There was a little bit of an arm race between ad blockers and advertisers in the middle of the 2010s, but it didn't last long. Advertisers mostly just decided not to care about ad-blockers.
>Advertisers mostly just decided not to care about ad-blockers.
Directly not to care because they lost in court.
And yet the biggest advertizer on Earth (Google) decided to change their browser to make adblocking far more difficult. That or they say "just use an app, oh and turn on notifications". I'm not exactly sure who you think won the arms race there, but it seems like we the user did not.
There is significantly more spam than 20 years ago, just less of it reaches your inbox. This is a very important distinction as the cost of spam filtering is just as high as ever. On top of that most people have given up on their own email servers and instead depend on Google/Microsoft to do it for them. This allows these companies to have an overwhelming influence on email on the internet, to the point they can send spam with near impunity, and where if your system does it will be nuked from orbit by their systems.
And much like now Google supplies both the email spam, and the solution to the spam, they'll gladly supply the LLMs spam and the LLM solution while applying their 'flavor' of what's allowed to the entire internet.
> Directly not to care because they lost in court.
I'm pretty sure the illegal sport streaming websites didn't stop doing that just because it became illegal, otherwise they could have stopped their activity altogether while they were at it…
> I'm not exactly sure who you think won the arms race there, but it seems like we the user did not.
I, at least, won when the webiste showing ads gave up the race (for the past decade at least, only time will tell about the future).
> nd yet the biggest advertizer on Earth (Google) decided
This is actually an argument in my direction! The owners of websites (which are also the ones posting slop today) didn't care enough and the situation only changed because Google moved.
I expect the same thing with slop. Individual websites won't make any effort to make their slop unblockable, and it will only be a problem if OpenAI/Anthropic/Google decide that they care about this market. But unlike Google in the ads market, I don't think the model providers have any reason to care. The web is already dead in their mind anyway.
> There is significantly more spam than 20 years ago, just less of it reaches your inbox.
This goes against your very argument from earlier!
> On top of that most people have given up on their own email servers and instead depend on Google/Microsoft to do it for them.
Out of convenience, but you don't need that to be practically free of spam. Whatever version of SpamAssassin is being run on OVH's mail servers has been enough for that purpose for me.
> they'll gladly supply the LLMs spam and the LLM solution while applying their 'flavor' of what's allowed to the entire internet.
Again, they don't care about the web. They just crawl it for content but they don't want you to read any webpage, they want you to stay in their chatbot. Every other use-case is nonexistent to them (except coding agents, of course, but that's a different product altogether).
It does mean that this will have a drift problem if it's just trained on the idiosyncrasies of model fine tuning. That's fine! But it is something to be aware of.
"Text is simply not information dense enough to be able to decode some arbitrary signal of provenance from it...it's a bad fiction to perpetuate that any of this is anything more than tarot card reading."
Not true at all. Pangram is highly effective and has a very low false positive rate.
The post here is impressive for a small project, it looks like they independently thought of one of the core ideas Pangram uses of creating twins to compare.
You can see how it works here: https://arxiv.org/pdf/2402.14873
So, if the decision from Pangram determined, on every assignment, if you would be expelled from university for plagiarism, would that be acceptable to you regardless of how you actually did the work?
If you would not be okay with that, what level of consequence would be acceptable for the output from this tool?
That’s a different point.
I’d want detectors to be as accurate as possible, false positives of 1 in 10000 seems like a good starting point. I believe their results have been independently tested.
And as a separate matter, any tool for evaluating students should be applied fairly, safely, and with adequate human review and due process.
You need good tools and good oversight.
Agreed, that's a fair and reasonable stance.
The reason I asked is that I have a hard time understanding the point of these tools. When it comes to education, it can be a matter of learning objectives. But outside that, what's the point?
The prediction from the tool is pointless for deciding on copyright or contract issues, and other text should be judged on its correctness or applicability to the task.
If all the tool is good for is "maybe this student cheated, but only an in-depth investigation would maybe prove it", it isn't a very useful tool, because it's more straightforward to just mandate that evidence is submitted regardless of what the tool says. On top of that, even the lack of evidence of manual work isn't good proof of using LLMs.
Due process should never just become a checkbox item. To deal with lives and livelihoods justly, you need appeal pathways and meaningful liability exposure for the processors.
Plagiarism and cheating sucks for everyone. Worth solving.
pow(n,m) where n is alphabet size and m is number of characters is very dense.
i think one thing overlooked by this perspective is that many of a detectors adversaries are not that sophisticated. so despite this i think it is a useful thing to try to do. particularly when people are trying to do fraud which will often having to use abliterated models and generally trying to be as economical in their efforts
I don't know, the thing about most text slop is how little effort goes into disguising it (for now, anyway). I'm sure anyone dedicated can go undetected, but it's the really low-effort stuff that's generally the problem. If you can catch some of it, that's something at least.
This sounds like it was edited by an llm.
Sure it is; we do it all the time, and then we modify each other's etc, etc; english we speak today was spoke yesterday waspake the same in yesteryears; we have no trouble dating english or other languages to a time.
A better argument is people themselves are just too influenced by reading that they'll sound like LLMs in a couple of years.
It depends on how much text. For example, chardet often falls down on short strings, but 1K characters it nails it.
The best method is, as always, an anti-privacy method.
Simply track all citizens' writing patterns throughout their life, from cradle to grave, then diff with any given text's signature--you'll know if it was human written or not.
Better--opt in--install a "personal text signature" on your devices, sign things that you wrote yourself with it.
But I suppose that's just like the image provenance chips on cameras.
Either way father fascism is more with us than ever, praise him!