I always got voted down when I posted the evaluation of the parent articles I got from my Ouija board. I just want to help people understand whether they should just reject bad articles, without having to bother reading them.
I'm moving on to evaluating articles with a modified lie detector test and tarot cards, I'm sure that'll help my credibility and give my public rejections more authority.
Do you have evidence Pangram is unreliable? There are independent evaluations [1, 2] showing it works, and it's getting used more and more scientific papers. Have you used it or evaluated it yourself? What do you think these other evaluators are doing or getting wrong?
I don't doubt that those detectors are generally correct. Pangram seems to be particularly accurate. I see independent evaluations ranging from 97% accurate to over 99%. Frankly, I somewhat doubt those numbers, but I do agree that LLM usage can be fairly accurately detected.
But procedurally, there are huge issues involved with automated tools used to harm other people. You are one of the 0.5% percent of people whose article was flagged as LLM-generated when it wasn't, one of the false positives. What do you do? Argue? The accusers will claim that you're 99.5% likely to be lying.
It's the same issue we have with automated customer service, automated insurance claims, and so forth. It is usually correct, and terrifically unjust when it fails... at which point there is no recourse. In a perverse sense, its accuracy can be a drawback, because if the false positive rate low enough, nobody is going to believe you when you're falsely accused. And people will be falsely accused.
I think it's ironic that it seems like it capitalizes on the same flaw that most LLM-posting does... "Chat GPT is usually right, I'm going with it." You shouldn't post an LLM article without independently validating its claims, so that there is a responsible person in the loop. The same is true for rejections and accusations, but more so, because they're more damaging.
I always got voted down when I posted the evaluation of the parent articles I got from my Ouija board. I just want to help people understand whether they should just reject bad articles, without having to bother reading them.
I'm moving on to evaluating articles with a modified lie detector test and tarot cards, I'm sure that'll help my credibility and give my public rejections more authority.
Do you have evidence Pangram is unreliable? There are independent evaluations [1, 2] showing it works, and it's getting used more and more scientific papers. Have you used it or evaluated it yourself? What do you think these other evaluators are doing or getting wrong?
1: https://bfi.uchicago.edu/insights/artificial-writing-and-aut... 2: https://arxiv.org/pdf/2501.15654
I don't doubt that those detectors are generally correct. Pangram seems to be particularly accurate. I see independent evaluations ranging from 97% accurate to over 99%. Frankly, I somewhat doubt those numbers, but I do agree that LLM usage can be fairly accurately detected.
But procedurally, there are huge issues involved with automated tools used to harm other people. You are one of the 0.5% percent of people whose article was flagged as LLM-generated when it wasn't, one of the false positives. What do you do? Argue? The accusers will claim that you're 99.5% likely to be lying.
It's the same issue we have with automated customer service, automated insurance claims, and so forth. It is usually correct, and terrifically unjust when it fails... at which point there is no recourse. In a perverse sense, its accuracy can be a drawback, because if the false positive rate low enough, nobody is going to believe you when you're falsely accused. And people will be falsely accused.
I think it's ironic that it seems like it capitalizes on the same flaw that most LLM-posting does... "Chat GPT is usually right, I'm going with it." You shouldn't post an LLM article without independently validating its claims, so that there is a responsible person in the loop. The same is true for rejections and accusations, but more so, because they're more damaging.
because it's not much better than an RNG?
What data supports that conclusion about Pangram?
Sad little world we live in tbh.