I’m more optimistic about the possibility of beneficial AGI in general than most folks, I think, but something that caught me in the article was the recourse to mammalian sociality to (effectively) advocate for compassion as an emergent quality of intelligence.
A known phenomenon among sociologists is that, while people may be compassionate, when you collect them into a superorganism like a corporation, army, or nation, they will by and large behave and make decisions according to the moral and ideological landscape that superorganism finds itself in. Nobody rational would kill another person for no reason, but a soldier will bomb a village for the sake of their nation’s geostrategic position. Nobody would throw someone out of their home or deny another person lifesaving medicine, but as a bank officer or an insurance agent, they make a living doing these things and sleep untroubled at night. A CEO will lay off 30,000 people - an entire small city cast off into an uncaring market - with all the introspection of a Mongol chieftain subjugating a city (and probably less emotion). Humans may be compassionate, but employees, soldiers, and politicians are not, even though at a glance they’re made of the same stuff.
That’s all to say that to just wave generally in the direction of mammalian compassion and say “of course a superintelligence will be compassionate” is to abdicate our responsibility for raising our cognitive children in an environment that rewards the morals we want them to have, which is emphatically not what we’re currently doing for the collective intelligences we’ve already created.
> Nobody rational would kill another person for no reason, but a soldier will bomb a village for the sake of their nation’s geostrategic position.
I think you're forgetting to control for the fact that the former would be severely punished for doing so, and the latter would be severely punished for not doing so?
> Nobody would throw someone out of their home or deny another person lifesaving medicine, but as a bank officer or an insurance agent, they make a living doing these things and sleep untroubled at night.
Again, you're forgetting to control for other variables. What if you paid them equally to do the same things?
Why should you "control" for these variables? AIs will effectively be punished for doing various inscrutable things by their own internal preferences.
> I think you're forgetting to control for the fact that the former would be severely punished for doing so, and the latter would be severely punished for not doing so?
That’s interesting and I think it’s more complicated. Here are some half-finished thoughts:
I imagine a grunt soldier would indeed be more likely to follow an order to nuke the world than a general would be to issue the order/push the button- and part of this is because the punishment for the grunt would be much greater, where the general is afforded more latitude in decision making.
However, the grunt may have volunteered to submit to the potential punishments, having signed a contract with the army. He made a choice in that regard.
If you want to be able to make your own decisions (e.g. choose NOT to drop the bomb when ordered) you have to have power to defend against “punishment” or unwanted consequences imposed by others. For a grunt, this might look like physical ability to defend themselves (2nd amendment comes to mind) , or economic independence via a homestead, or something else.
Interesting to think about.
> I think you're forgetting to control for the fact that the former would be severely punished for doing so, and the latter would be severely punished for not doing so? > What if you paid them equally to do the same things?
I think the larger point is that rewarding bombing, or paying bank officers to evict people from their homes is how the superorganism functions. Your counter examples are like saying 'what if fire was cold instead of hot', well then it wouldn't be fire anymore.
I dispute that? There are plenty of e.g. countries that don't bomb others, especially not for "no reason". (!) And the whole point here was about individuals behaving differently when part of the collective, not about the collective having been set up with different incentives and rules than the individuals were in the first place. You can have collectives with better incentives set up and achieve more humane outcomes. Like I said, such examples really exist, they're not hypothetical.
Show me a country that doesn’t bother other countries — ever — and I’ll show you a country that doesn’t have any cards to play. Except for maybe isolated island nations who lack the ability to threaten anyone, all nations come into conflict with others and the only ones that “don’t [initiate aggression with] others” are the ones who lack the ability or who have done the calculation that they’d be severely slapped back if they tried, so they wisely don’t poke the bear(s).
Well said. No organism willingly commits perceived suicide unless it's a viable strategy for its continued existence. The reason a thing exists, is because it hasn't tempted its potential predator.
>>> Nobody rational would kill another person for no reason, but a soldier will bomb a village for the sake of their nation’s geostrategic position.
>> There are plenty of e.g. countries that don't bomb others, especially not for "no reason". (!)
> Show me a country that doesn’t bother other countries — ever
Do you by any chance happen to feel like you may have moved the goalposts by at least a tiny inch?
To use a quote from of those large corporate leaders (Warren Buffett & Charlie Munger):
"Show me the incentive and I will show you the outcome"
It is carrots and sticks all the way down...
There is no human superoganism, and the reason we’re doomed as a temporary species is precisely that humans cannot act eusocially as a superorganism.
By your definition the Moscow Metallica show, Jan 6th riots, etc… were superorganisms and that’s not even barely applicable
Humans expressing group behaviors at some trivial number for a trivial period (<1M people for <2 days is the largest sustained group activity I’m aware of) is the equivalent of a locust swarm not even close to a superorganism
a CEO laying off 3% scales in absolute numbers as the company grows
should, therefore, large companies, even ones that succeed largely in a clean way by just being better at delivering what that business niche exists for, be made to never grow too big, in order to avoid impacting very many people? keep in mind that people engage in voluntary business transactions because they want to be impacted (positively—but not every impact can be positive, in any real world)
what if its less efficient substitutes collectively lay off 4%, but the greater layoffs are hidden (simply because it's not a single employer doing it which may be more obvious)?
to an extent, a larger population inevitably means that larger absolute numbers of people will be affected by...anything
I think it's reasonable that bigger companies are under more scrutiny and stricter constraints than smaller companies, yeah.
Keeps actors with more potential for damaging society in check, while not laying a huge burden on small companies which have less resources to spend away from their core business.
> voluntary business transactions
The evil parts are hid in property rights which are not voluntary.
> made to never grow too big, in order to avoid impacting very many people
Consolidated property rights have more power against their counterparties, that's why businesses love merging so much.
Look at your tax return. Do you make more money from what you do or what you own? If you make money from what you do, you're a counterparty and you should probably want to tap the brakes on the party.
What are the evil parts, exactly? When property can't be privately owned with strong rights the effectively the government owns everything. That inevitably leads to poverty, often followed by famine and/or genocide.
Plenty of examples on both sides of that, even in the US there’s vast swaths of land that can’t be privately owned for example try and buy a navigable river or land below the ordinary high water mark etc. https://en.wikipedia.org/wiki/Navigable_servitude Similarly eminent domain severely limits the meaning of private land ownership in the US.
The most extreme capitalist societies free from government control of resources like say Kowloon Walled City are generally horrible places to live.
Why is it that property is taxed less than productive work? Someone sitting on their ass doing nothing but sucking resources through dividend payments has that income taxed less than the workers' income who did the work that generated those dividends. Why isn't the reverse the case? Heavily tax passive income, and lightly tax active income. Incentivize productive activity and penalize rent-seeking parasites?
The reason to not tax investment super harshly is that it’s an incentive to not just stuff your money under a mattress. Many people have jobs and careers they wouldn’t otherwise have specifically because 100 rich guys stuck their money in various VC funds (especially in this here audience on HN). If we taxed the idea of investment ruinously, we decrease that incentive. You may think that somehow all those jobs (or more) would somehow materialize without investors investing, but that’s a hypothesis or an argument, not a proven conclusion.
I've heard this argument, but as long as there is some return then people will invest because some is better than none. VC investments have the potential to return insane amounts, so people will still buy those lottery tickets even if the profits are taxed. I know this because actual lottery winnings are taxed and people still buy those tickets in huge numbers. And if you are saying that taxing investments the same as worked income is 'super harshly' and 'ruinously' high, then what does that say about the state of wage taxes?
Places with predominantly private ownership can be and are prone to famine, and/oril genocide, etc. as well.
Sure, but those are the exceptions that prove the rule. Centralized (Marxism and its descendants) societies tend to have those things happens the majority of the time. In decentralized capitalist societies, they happened once a long time ago and we took steps for them to not happen again. Seems like a flaw in those societies is that when these problems happen so infrequently people forget and then you get takes like this.
I think that Marxism is not centralized - Capitalism is centralized and some communist implementations are centralized. Marxism, if anything, is distributed or communal.
It's not "nobody owns anything", it's "everybody owns everything". Maybe those mean the same thing to some people, but that's the idea.
Exactly this. Lenin's whole thing was to reach a point where the sate dissolves!
It drives me crazy how like 95% of HN is too scared or lazy to read a 10min wikipedia article on Marxism.
It’s your position that we just don’t get how genius Marxism is and that it just hasn’t been tried? Why did it not work out so well for the USSR? When was that state going to “dissolve” into a utopia where everyone owns everything? Why was China a poor agrarian society when they followed Marxism better, and has become relatively wealthy since abandoning a great deal of those ideas and participating in a form of capitalism?
I think it's not exactly a fair comparison, because capitalism and communism are both very new economic systems. And, in the time communist systems existed, they were existentially threatened by high GDP nations.
I think, it's clear to me, that capitalists feel extremely threatened by the mere concept of Marxism and what it could mean for them. Even if it's happening on the other side of the world. They will deploy bombs, soldiers, develop nukes.
I'm not saying that it works and it's good. But, consider: most capitalist nations are abject failures as well. There's only a handful of capitalist nations that are developed, and they stay developed because they imperialisticly siphon wealth from the global periphery. We don't know if this system is sustainable. Really, we don't.
Since WWII, the US has just been riding the waves of having 50% of the global GDP. It's not that we're doing good - it's that everyone else was bombed to shreds and we weren't. We've sort of been winning by default. I don't think that's enough to just call it quits.
Centralised planning is not what Marxism is about though, Marxism is about class struggle and the abolishment of a capital-owning class, distributing the fruits of labour to the labourers.
In that definition it's even more decentralised than capitalism which has inherent incentives for the accumulation of capital into monopolies, since those are the best profit-generating structures, only external forces from capitalism can reign into that like governments enforcing anti-trust/anti-competitive laws to control the natural tendency of monopolisation.
If the means of production were owned by labourers (not through the central government) it could be possible to see much more decentralisation than the current trend from the past 40 years of corporate consolidation.
The centralisation is already happening under capitalism.
Yep, a common US example of Marxism is when Farmer owned co-ops for collecting and distributing crops. That model is well aligned with protecting family farms by avoiding local rent seeking monopolies.
Other parts of the agro sector are far more predatory, but it’s hard do co-op style manufacturing of modern farm equipment etc. Marxism was created in a world where Americans owned other Americans it’s conceptually tied into abolitionist thinking where objecting to the ownership of the more literal means of production IE people was being reconsidered. In that context the idea of owning farmland and underpaying farm labor starts to look questionable.
Unfortunately you’re taking to the void here
People can’t differentiate between what Marx wrote and what classic dictators (Lenin, Stalin, Mao) did under some retcon “Marxist” banner
So you never read what Marx wrote then I take it. His ideas were even more unworkable than what the Communists tried. For example, he didn't understand specialization and thought people could just change jobs each day based upon whim. This is a big reason why the Marxists have never been able to convince the working class and why their support always came from the bureaucracy and not from people who actually did the work.
I understand it perfectly
I don’t agree with it for more fundamental reasons than you describe
Namely that he was trying to apply Hegelian dialectic with political philosophy when the dialectic is an empirical dead end mathematically so could never even theoretically solve the problems he was pressing on
Don’t confuse understanding with agreement
> Centralised planning is not what Marxism is about though
What an incredibly dishonest thing to say. Go to a former Communist country and tell them this. They will either laugh you out of the room, or you will be running out of the room to escape their anger.
They can laugh all they want, I understand their resentment from being oppressed into a failed experiment which misused the "marxist" label to propagandise itself. Still doesn't mean that Marxism is about centralised planning though.
Indeed, by what moral justification does one slow the wheels of commerce, no matter how many people they run over?
[dead]
Beautifully expressed.
> Nobody would throw someone out of their home or deny another person lifesaving medicine
Individuals with rental properties and surgeons do this every day.
Quibble, surgeons are not the ones doing this. Surgeons' schedules are generally permanently full. They do not typically deny people lifesaving medicine, on the contrary they spend all of their time providing lifesaving medicine.
The administrators who create the schedule for the surgeons, are the one denying lifesaving care to people.
Triage, whether by overworked nurses or by auction or by private death panel or by public death panel, is not necessarily a problem created by administrators. It can be created by having too few surgeons, in which case whatever caused that (in a time of peace, no less) is at fault. Last I heard it was the doctor's guild lobbying for a severe crimp on their training pipeline, in which case blame flows back to some combination of doctors and legislators.
You heard wrong. While at one point the AMA lobbied Congress to restrict residency slots, they reversed position some years back. However Congress has still refused to increase Medicare funding for residency programs. This is essentially a form of care rationing imposed through supply shortages.
https://savegme.org/
There is no "doctor's guild". No one is required to join the AMA to practice medicine, nor are they involved in medical school accreditation.
Like I said, some combination of doctors and legislators. If doctors lobbied the laws (or budgetary line items) onto the books and they are still in effect, they still have culpability.
Blaming congress too is fine, but let's be clear: someone has to fight to increase every budget and the AMA didn't just know this when they were structuring their proposal, didn't just count on it not happening, they considered this an implementation detail subordinate to the openly admitted primary objective of propping up physician wages as the Greatest Generation passed. That was always the goal, they were extremely open about it, and about 15 years ago I was attending a talk on demographics in medicine with a primarily physician audience, one of them asked what the plans were to change this to staff up for the Boomer wave (the bump was on the slide, begging the question) and the presenter waved his hand and said maybe they could do something... or not, and then he laughed, and the rest of the room laughed with him.
I'm glad that the AMA has changed their stated position now that it's too late to change course (for the Boomers anyway) and their squeeze is bearing fruit for them and suffering for their patients, but I'll always remember that room full of doctors and doctors-to-be laughing about the prospect of intentionally understaffing for profit. I have it filed in my memory right next to the phone call of Enron traders giggling as they ordered power plants offline to scare up prices, except it's about a million times worse.
I'm not even talking about triage. It's not a matter of who has the worst problem, it's about which patient the nurses deliver to the surgeon and anesthesiologist. Literally just who gets scheduled and when.
If all of the surgeons' schedules are full, the administrators are as innocent as the surgeons.
If the surgeons are busy each day, that removes all responsibility for who gets added to their schedule 3 months in advance? Please elaborate.
Surely they could volunteer to do some charity surgery in their own time. They aren't slaves.
Sure! They can volunteer:
- Their skills.
- Their time.
- The required materials to properly perform the surgery.
They can't volunteer:
- The support staff around them required to do surgery.
- The space to do the surgery.
Surgery isn't a one-man show.
What did you mean by "Surely they could volunteer to do some charity surgery in their own time. They aren't slaves?"
There are a lot of individuals who have the ability to provide those resources.
Even if that's a bad example, there are innumerable examples where individuals do choose not to help others in the same way that corporations don't.
Frankly, nearly every individual is doing that by not volunteering every single extra dollar and minute they don't need to survive.
You've now turned a moral willingness-to-help problem into a logistical and coordination problem.
What you suggest requires entire organizations to execute properly. These organizations do exist, such as Doctors Without Borders.
I don't think your original claim is fair, which amounts to "any surgeon who does not participate in Doctors Without Borders is just as bad as a landlord who evicts a family during winter".
What do you think we owe to one another, philosophically?
It's not about what I think. The post I replied made the assertion that individuals don't turn people away like corporations do (essentially).
My point is that individuals choose not to help others constantly. Every time I see a homeless person, I don't offer them a couch to sleep on. I could, at least once, but I don't. We all do that, most days multiple times.
And yes, that does apply to doctors who don't volunteer services. It applies to me too and, I bet, to the OP as well.
Firstly, there's a difference between failing to take an action, such as not offering a homeless person a couch, and actively taking an action, such as kicking someone out of their home.
Secondly, as discussed, the "individuals don't turn people away, corporations do" dynamic really does apply to doctors. If you were, say, on an airplane with a doctor sitting next to you, and you managed to cut yourself or burn yourself or something, I would bet they would render aid.
Basically you're equating turning someone away, and withdrawing something that someone has, with failing to actively seek out people who could need help. But I don't think those are morally equivalent. Maybe you're a utilitarian and that's fine, but I'm a virtue ethicist and I do not agree that equality of outcome means equality of morality.
Not really, because surgeons require operaing rooms and support staff and equipment to do what they do, all of which are controlled bybthe aforementioned hospital administrators.
Yeah, it's the natural empathy myth. Somebody totally would kill somebody else for some reason. It's not inherent to being human that you're unable to be steely-hearted and carry out a range of actions we might classify as "mean" - and those mean actions can have reasons behind them.
So, OK, abdication of responsibility to a collective is a thing. Just following orders. So what? Not relevant to AGI.
Oh wait, this is about "superintelligence", whatever that is. All bets are off, then.
The superintelligence might decide based on things only it can understand that the existence of humans prevents some far future circumstance where even more "good" exists in the universe. When it orders you to toss the babies into the baby-stomping machine, perhaps you should consider doing so based on the faith in its superintelligence that we're supposed to have.
Human beings aren't even an intelligent species, not at the individual level. When you have a tribe of human beings numbering in the low hundreds, practically none of them need to be intelligent at all. They need to be social. Only one or two need to be intelligent. That one can invent microwave ovens and The Clapper™, and the rest though completely mentally retarded can still use those things. Intelligence is metabolically expensive, after all. And if you think I'm wrong, you're just not one of the 1-in-200 that are the intelligent individuals.
I've yet to read the writings of anyone who can actually speculate intelligently on artificial intelligence, let alone meet such a person. The only thing we have going for us as a species is that, to a large degree, none of you are intelligent enough to ever deduce the principles of intelligence. And god help us if the few exceptional people out there get a wild bug up their ass to do so. There will just be some morning where none of us wake up, and the few people in the time zone where they're already awake will experience several minutes of absolute confusion and terror.
And lenders and insurers.
I would argue that corporate actors (a state, an army or a corporation) are not true superorganisms but are semi-autonomous, field-embedded systems that can exhibit super-organism properties, with their autonomy being conditional, relational and bounded by the institutional logics and resource structures of their respective organisational fields. As the history of humanity has shown multiple times, such semi-autonomous with super-organism properties have a finite lifespan and are incapable of evolving their own – or on their own – qualitatively new or distinct, form of intelligence.
The principal deficiency in our discourse surrounding AGI lies in the profoundly myopic lens through which we insist upon defining it – that of human cognition. Such anthropocentric conceit renders our conceptual framework not only narrow but perilously misleading. We have, at best, a rudimentary grasp of non-human intelligences – biological or otherwise. The cognitive architectures of dolphins, cephalopods, corvids, and eusocial insects remain only partially deciphered, their faculties alien yet tantalisingly proximate. If we falter even in parsing the intelligences that share our biosphere, then our posturing over extra-terrestrial or synthetic cognition becomes little more than speculative hubris.
Should we entertain the hypothesis that intelligence – in forms unshackled from terrestrial evolution – has emerged elsewhere in the cosmos, the most sober assertion we can offer is this: such intelligence would not be us. Any attempt to project shared moral axioms, epistemologies or even perceptual priors is little more than a comforting delusion. Indeed, hard core science fiction – that last refuge of disciplined imagination – has long explored the unnerving proposition of encountering a cognitive order so radically alien that mutual comprehension would be impossible, and moral compatibility laughable.
One must then ponder – if the only mirror we possess is a cracked one, what image of intelligence do we truly see reflected in the machine? A familiar ghost, or merely our ignorance, automated?
> I would argue that corporate actors (a state, an army or a corporation) are not true superorganisms but are semi-autonomous, field-embedded systems that can exhibit super-organism properties, with their autonomy being conditional, relational and bounded by the institutional logics and resource structures of their respective organisational fields.
Lotsa big words there.
Really, though, we're probably going to have AI-like things that run substantial parts of for-profit corporations. As soon as AI-like things are better at this than humans, capitalism will force them to be in charge. Companies that don't do this lose.
There's a school of thought, going back to Milton Friedman, that corporations have no responsibilities to society.[1] Their goal is to optimize for shareholder value. We can expect to see AI-like things which align with that value system.
And that's how AI will take over. Shareholder value!
[1] https://www.nytimes.com/1970/09/13/archives/a-friedman-doctr...
Costs will go down. But so will revenue, as fewer customers have an income because a different company also cut costs.
Record profits. Right up until the train goes off a cliff.
That assumes that consumers will just accept it. I would not do business with an AI company, just as I don’t listen to AI music, view AI pictures or video, or read AI writings. At least not knowingly.
People would absolutely buy AI farmed meat or vegetables if they were 10% cheaper. The number of people who pay a premium depending on production method is a small minority.
As long as you stay inside capitalism you unquestionably and unequivocally do business with an AI company.
I mean, I don't know of any companies that are purely digital, with an AI as the CEO. Maybe that's coming, but I would not intentionally buy anything from such a company, or work for such a company, or invest in such a company.
I guess it would be like being a vegan. It might be a pointless effort in the grand scheme of things, but at least I can say that I am not contributing.
So OpenAI isn’t an AI company then?
Also sociopaths are more capable of doing those things while pretending they are empathetic and moral to get positions of power or access to victims. We know a certain percentage of human mammals have sociopathic or narcissistic tendencies, not just misaligned groups of humans that they might take advantage of by becoming a cult leader or war lord or president.
> soldier will bomb a village for the sake of their nation’s geostrategic position.
Soldier does that to please the captain, to look manly and tough to peers, to feel powerful. Or to fulfill a duty - moral mandate on itself. Or out of hate, because soldiers are often made to hate the ennemies.
> Nobody would throw someone out of their home or deny another person lifesaving medicine
They totally would. Trump would do it for pleasure of it. Project 2025 authors would so it happily and sees the rest of us as wuss. If you listen to right wing rhetorics and look at voters, many people will hapilly do just that.