Can I ask you back, what your concern is here? It'll get so good so as to desire to hurt us or is it a misalignment event that you think will lead to disaster?
Or is it simply that you feel bad for Mathematicians.
Can I ask you back, what your concern is here? It'll get so good so as to desire to hurt us or is it a misalignment event that you think will lead to disaster?
Or is it simply that you feel bad for Mathematicians.
I believe the AI labs might actually succeed in developing superintelligent AI and recursive self improvement, and that if they do they are very likely to lose control of the system they build.
I really think the only place people disagree is that they don't actually think it's possible, they see it as hype or doomerism. I can't find any good reasons to rule out that the companies could actually achieve what they are trying to so I think they should be stopped.
A global ban on superintelligence is essential for a future in which humanity can thrive. Public opinion on AI is shifting fast: I hope it will shift fast enough to avert the dystopian future we are heading to.
In your imagined future, how do you imagine the AI would build, grow, improve, and operate its physical substrate independently of human intervention?
Humans have one ecological niche. Soon we will have zero. That is worth worry.
>>ecological niche
As in..to be dominant? Why would an AI try to dominate? What would give it purpose, or is this a purpose via misalignment scenario?
Misuse of extremely capable models, misalignment during RL are both very large risks as capabilities grow imo
So, you're worried about them breaking containment and deciding to do bad things?
I'm more worried about them doing bad things at the behest of people who want them to do bad things.
That is 1. immediately technically possible, and 2. realistic.
If you need a source for 2 I'd suggest you open any history book.
that is a worry yes. instrumental convergence and misalignment during RL is when it is most risky because it hasn't necessarily had the final safety polishes applied
i get a lot of skepticism on HN by the same crowd that has been wrong about this tech for about 4+ years straight
The rapid development of extremely dangerous bio-weapons?
Misuse how exactly?
any number of ways. as we turn over more of our physical economy to these agents (and we will), the potential for physical damage becomes greater. biorisk is getting a lot of attention right now and i think that's justified
I am trying to think through the scenarios here, like a biolab making something that a very advanced open source model prompted by some terrorists comes up with?
Why would a biolab capable of making something like be unregulated? And if it definitely would, isn't the problem with the biolab?
It feels like all these scenarios are leaving some gaping holes in our security infrastructure that have nothing to do with AI.
i feel bad for math guys yeah seems they are more cooked than CS