Once men turned their thinking over to machines
in the hope that this would set them free.
But that only permitted other men with machines
to enslave them.
...
Thou shalt not make a machine in the
likeness of a human mind.
-- Frank Herbert, Dune
You won't write, except prompts for your LLM. Why write code or prose when the machine can write it for you?
You won't think or analyze or understand. The LLM will do that.
This is the end of your humanity. Ultimately, the end of our species.
Currently the Poison Fountain (an anti-AI weapon, see https://news.ycombinator.com/item?id=46926439) feeds 2 gigabytes of high-quality poison (free to generate, expensive to detect) into web crawlers each day. Our goal is a terabyte of poison per day by December 2026.
Join us, or better yet: deploy weapons of your own design.
You shouldn't take a sci-fi writer's words as a prophecy. Especially when he's using an ingenious gimmick to justify his job. I mean, we know that it's impossible for anyone to tell how the world will be like after the singularity, by the very definition of singularity. Therefore Herbert had to devise a ploy to plausibly explain why the singularity hadn't happened in his universe.
If only we could look into the future to see who is right and which future is better so we could stop wasting our time on pointless doomerism debate. Though I guess that would come with its own problems.
Hey, wait...
I like the idea that Frank Herbert’s job was at risk and that’s why he had to write about the Butlerian Jihad because it kind of sounds like on the other side you have Ray Kurzweil, who does not have to justify his job for some reason.
Does seem funny to think of sci fi writers as being particularly concerned about justifying their jobs.
If you read this through a synth, you too can record the intro vocal sample for the next Fear Factory album
I call your Frank Herbert machine dystopia and raise you the Ian Banks machine utopia...
I'm gonna call that raise: How does one get to the anarchist Culture when all the machines are being built by profit-hungry capitalists?
We build our own with data that we've collected ourselves ethically. Then we execute once the big guys are distracted.
I would bet a lot of money on your poison is already identified and filtered out of training data.
Like partial courses of antibiotics, this will only relatively-advantage thoae leading efforts best able to ignore this 'poison', accelerating what you aim to prevent.
yes. whoever has the best (least detectable) model is best poised to poison the ladder for everyone.
Looking through the poison you linked, how is it generated? It's interesting in that it seems very similar to real data, unlike previous (and very obvious) markov chain garbage text approaches.
We do not discuss algorithms. This is war. Loose lips sink ships.
We urge you to build and deploy weapons of your own unique design.
[flagged]
It's based on this paper BTW: https://www.anthropic.com/research/small-samples-poison
[flagged]
>Why write code or prose when the machine can write it for you?
I like to do it.
>You won't think or analyze or understand. The LLM will do that.
The clear lack of analysis seems to be your issue.
>This is the end of your humanity. Ultimately, the end of our species.
Doubtful.
"The end of humanity" has been proclaimed many times over. Humanity won't end. It will change like it always has.
We get rid of some problems, and we get a bunch of new problems instead. And on, and on, and on.
Russell's chicken (or turkey) would like a word.
https://en.wikipedia.org/wiki/Turkey_illusion
I love that you brought this up.
Chickens are killed ALL the time. It’s a recurring mass event. If you were a smart chicken you could see that pattern and put it into a formula.
In contrast, the end of Humanity would be a singular event. It’s even in the name…
And that is fiction / speculation in comparison. It’s not backed by any data. Human survival over 300,000 years by contrast is.
I mean it’s fine to dream things up, but let’s be fair and call it what it is.
On the other hand, species go extinct with increasing regularity.
What, you weren't alive when the last mass extinction event occurred? Why didn't you communicate or at least write the last handful down or something? Aren't you smarter than a chicken?
It's funny that you think we know what happened to humans anymore than a chicken knows what happened to chickens.
One thing I've wondered about is:
Suppose a civilization (but not species) ending event happens.
The industrial revolution was fueled (literally) by easy-to-extract fossil fuels. Do we have enough of those left to repeat the revolution and bootstrap exploitation of other energy sources?
298,000 of those years didn't have toilet paper. It was utterly impossible for a single person to "end humanity" even 200 years ago; now, the president can do it in minutes by launching a salvo of nukes. Comparing the present moment to the hunter/gatherer days is preposterous.
It’s absurd and not scientific to claim that "a salvo of nukes" will kill humanity.
We don’t know how this will play out. It never happened before. Same with the chicken above.
For pretty much every single person you or I personally know, that would be the equivalent of the end of humanity.
Let’s not nitpick here. Worldwide human suffering and tragedy is equivalent to the end of humanity for most.
We can sit here and armchair while in the most prosperous, comfortable era of human history. But we also have to recognize that this era is a blip of time in history. That is a lot of data showing humanity surviving sure. But it’s also a very small amount of data showing any kind of life most would want to live in.
It only has to be right once. Humanity won’t end until it does.
Humanity may end if someone else goes to the top of food chain.
Bold of you to assume people will be writing in any form in the future. Writing will be gone, like the radio and replaced with speaking. Star Trek did have it right there.
>You won't read/write/think/understand etc...
I can't see it. We have LLMs now and none of that applies to me. I find them quite handy as a sort of enhanced Google search though.
[dead]
I think you’re missing the point of Dune. They had their Butlerian Jihad and won - the machines were banned. And what did it get them? Feudalism, cartels, stagnation. Does anyone seriously want to live in the Dune universe?
The problem isn’t in the thinking machines, it’s in who owns them and gets our rent. We need open source models running on dirt cheap hardware.
The point of Dune is that the worst danger are people who obey authority without questioning it.
Then wouldn't open source models running on commodity hardware be the best way to get around that? I think one of the greatest wins of the 21st century is that almost every human today has more computing power than the entire US government in the 1950s. More computer power has democratized access and ability to disperse information. There are tons of downsides to that which we're dealing with but on the net, I think it's positive.
Does it also means the US government has x1000000 more power than the one in 1950 ?
speaking strictly from an energy standpoint (power grid, megatons of warheads, etc).. it's probably close to that number.
It isn't a way around, you still obey. Only now, the authority you obey is a machine.
That's not the point of Dune. Who blindly obeyed who?
The Fremen followed a messianic figure into a galaxy-wide holy war because the Bene Gesserit seeded their culture with manufactured prophecy as a failsafe.
... which overthrowing the machines didn't stop. People just found another authority to mindlessly obey.
The “poison fountain” is just a little script that serves data supplied by… somebody from my domain? It seems like it would be super easy for whoever maintains the poison feed to flip a switch and push some shady crypto scam or whatever.
Lol. Speak for yourself, AI has not diminished my thinking in any material way and has indeed accelerated my ability to learn.
Anyone predicting the "end of humanity" is playing prophet and echoing the same nonsensical prophecies we heard with the invention of the printing press, radio, TV, internet, or a number of other step-change technologies.
There's a false premise built into the assertion that humanity can even end - it's not some static thing, it's constantly evolving and changing into something else.
A large number of people read a work of fiction and conclude that what happened in the work of fiction is an inevitability. My family has a genetically-selected baby (to avoid congenital illness) and the Hacker News link to the story had these comments all over it.
> I only know seven sci-fi films and shows that have warned about how this will go badly.
and
> Pretty sure this was the prologue to Gattaca.
and
> I posted a youtube link to the Gattaca prologue in a similar post on here. It got flagged. Pretty sure it's virtually identical to the movie's premise.
I think the ironic thing in the LLM case is that these people have outsourced their reasoning to a work of fiction and now are simple deterministic parrots of pop culture. There is some measure of humor in that. One could see this as simply inter-LLM conflict with the smaller LLMs attempting to fight against the more capable reasoning models ineffectively.
Now that you mention it, it is pretty strange to see HN users parroting other people’s thinking (sci-fi writers) like literal sub-sapient parrots, while simultaneously decrying the danger of machines turning people into sub-sapient parrots…
Following that logic… the closest problem would be literally inbetween their ears.
it's like all of san francisco has had a collective stroke
do... do the "poison" people actually think that will make a difference? that's hilarious.
Let the kiddies have their crusade
A better approach is to make AI bullshit people on purpose.
This is essentially just that. The idea is that "poisoned" input data will cause AIs that consume it to become more likely to produce bullshit.