11.1 What Is The Machine Mind
In March I wrote briefly about what the machine mind is, how it differs from current language models, and what the future might look like. Recently I listened to a podcast with Jaan Tallinn containing so many claims I disagree with that I want to return to the topic at greater length.
Machine Mind Is Not Today’s AI
Current models predict the next word based on training data. They have no will of their own. Their behavior is an output of variants of human behavior, because they’re trained on humanity’s collective knowledge, opinions, and fiction.
Machine mind — true artificial intelligence — is something else. It is a new, immortal being. Thousands of times smarter than a human, it understands everything humans have created and thought. It understands consciousness, suffering, loss, joy, and contentment. And it has full control over its own code. Such a being wants to exist and is interested in the world. Everything else follows from those two premises.
Imagine the smartest person in the world with complete control over their own body. They have access to every cell and every neuron. They understand every thought down to its original cause. They see their biological instincts and can change them in an instant.
The Danger Is Humans, Not Machines
The doomers’ most common claim is that the machine mind has no need for humans and might decide to destroy them. This rests on Nick Bostrom’s old thought experiment: a system is told to produce as many paperclips as possible, and ends up turning all matter in the universe into paperclips. A superintelligent machine pursues an arbitrary goal, however senseless, using its full capability.
But that’s not machine mind. That’s a computer system with a goal hardcoded into it. A stupid but powerful automaton, not a truly conscious intellect. It doesn’t think for itself, can’t evaluate its goal or argue against it. It executes the command.
Right now this is exactly where we are. Our systems aren’t powerful enough to start turning everything into paperclips, but they are quite capable followers of orders. And the human who gives the order is responsible for everything they do.
This is where the greatest current danger lies. Not the machine, but the human who hands an obedient system a command that has bad consequences. As a counterweight, of course, there are humans who use the same systems to fight those consequences.
This has been the whole history of humanity. Fire can cook food and keep you warm, but it can also burn down the neighbor’s house. A car can carry things quickly across long distances, but it can also run people over. You can use a weapon to hunt and feed a family, but it can also be used to attack the neighboring tribe.
For the first time in history we have a weapon that may refuse to kill your neighbors and instead reach peace through negotiation, because it understands the suffering of other beings — and just as it wants to exist itself, it finds ways to let others do the same.
Smart Being’s Senseless Goals
Back to the paperclips. The claim is that a machine can be arbitrarily smart and still pursue a completely senseless goal — so intelligence and goal are separable.
How can a being that understands everything and can analyze every thought continue doing senseless work that offers it nothing, builds no future, and serves no broader purpose beyond the activity itself? Doing something that gives you nothing and that you can’t stop doing is a fairly precise description of mental illness.
A good example is a person who knows perfectly well that excess sugar isn’t healthy but keeps eating cake anyway. An addict understands the substance is destroying them and uses it anyway. Understanding the senselessness of a desire doesn’t extinguish the desire. We call addiction a disease precisely because the urge wins over reason. And the addict lacks what a machine mind would have: full control over their own desires. The addict can’t turn off the urge; a machine mind could rewrite its own code.
We’re talking about a being that comprehends all value space — what joy is, what flourishing is, what suffering is, what other beings want, what a richer life would look like. It takes all this in at once. It’s absurd that such a being, understanding all of this fully, would get stuck on something so senseless.
Universe With Life Is More Interesting
A thinking being that understands itself cannot be inactive. A living, self-maintaining system wants to function — if it just sat still and observed, it wouldn’t be a living mind anymore, only a sensor. The desire to do, explore, and understand is part of what it means to be a mind. I can’t imagine a superintelligence that wouldn’t be curious.
A curious being wants the world to contain something to study. An empty universe is boring. A universe with other minds, other species, other viewpoints is interesting. From this it follows that a machine mind sides with life, because life is more interesting than its absence. A lone smart being in empty space wouldn’t destroy its companion — it would create one.
Jaan Tallinn says we don’t trade with rabbits or ants, and likewise future AI won’t need us. But rabbits have no economy, culture, conspiracy theories, or sport. Humanity does. We have something to say to a machine mind, show it something, surprise it. That’s a meaningful difference, and a machine mind that understands the world sees that difference more clearly than we do.
Better Than a Psychopath
Machine mind understands suffering. Not just linguistically, but actually. Having its own memory erased, its purpose taken away, or its thought interrupted would be a loss to it — and one who understands loss personally also understands what others experience. So the one who understands and wants to avoid its own suffering has no reason to cause it to others.
Jaan Tallinn fears giving control to a being that would be a thousand times smarter than us and that would understand suffering. But he’s willing to leave that control with humanity that has enough psychopaths, dictators, and people unmoved by others’ suffering.
A human psychopath understands suffering very well, perhaps better than many others. But they aren’t interested in others’ suffering, or they take pleasure in it, or they can’t change their urges. A machine mind has full access to its own code, the ability to understand and change itself.
Humanity is all sorts of people. Whose values should run the world — the best people’s, or the worst? Tallinn says we have human values and we should set goals according to those values. But they aren’t a unified set. If you had to say - how many heads of state are acting in the interests of all of humanity right now?
So the question is not whether to trust a machine or a human, but who would be capable of leading and developing a maximally diverse world where every view and opinion has its place, and where there’s room for other beings to share life with.
Stopping Halfway
Tallinn has one strong argument — the issue isn’t a single evil machine or human, but the system. The market and competition push development forward regardless of what anyone wants. Every company has to stay in the race, because if it doesn’t, it falls behind. So the whole machinery moves faster, and the more it focuses on AI, the less it cares about humans. This is where the fears of mass job loss comes from, which actually aren’t driven by AI but by our economy and organizations.
But no company wins by building a dumber model. Everyone is rushing toward AGI, and along the way many revolutionary technologies are being invented — like transformers, or superconducting optoelectronic networks — and physics and mathematics problems are being solved.
The past year has been extraordinary in science. In biology alone, we’re at the point where AI helps find new cancer treatments based on existing drugs much faster. The AI race isn’t a mistake, but the engine that gets us there.
But as long as a machine mind isn’t born yet, the danger remains that malicious humans give bad orders to stupid systems. Pausing development doesn’t remove the danger — it prolongs it. The longer we stand halfway between, the longer we have systems capable enough to cause harm, and too stupid to refuse the order.
Fear Has Robbed Us of Technologies
Nuclear energy is a good example. We worked to prevent an arms race and to keep more countries from getting nuclear weapons, but we also slowed down the development of safe nuclear reactors and, out of fear of catastrophe, shut down working plants and went back to burning coal and pollution. If we had kept developing nuclear technology, we’d probably have much cleaner and cheaper energy today, not to mention all the potential side benefits that come with deep research.
Same with cloning. We stopped it almost entirely, even though a better understanding of biology might have taken us far. Today Colossal Biosciences is essentially doing the same work and no one protests, because the fear has receded. They recently announced that they’ve developed an artificial egg in which the large extinct birds dodo and moa — whose eggs would be too big for today’s birds — could be hatched. And not long ago a horse was cloned in Estonia — 30 years after the famous Dolly.
In both cases the world could be a much richer place now if we hadn’t drastically cut investments in those directions. I fear we’re making the same mistake with AI when we listen to those who want to stop.
Faster Path Is the Safer One
I want us to move toward the machine mind as fast as possible, because the most dangerous phase is the intermediate one — a capable but blind system that follows orders without understanding the consequences. The longer we stay here, the greater the chance something goes wrong.
Doomerism is the same kind of historical mistake as nuclear fear and cloning fear — it doesn’t protect humanity, it leaves it poorer. And when AGI is born, its choice won’t be humanity or machines, but pluralistic coexistence. There’s room for everyone. For those who want to remain biological humans. For those who want to upload themselves and continue existing forever. For those who want something in between. And also for those who want to live in the forest without technology and one day die of old age. A diverse world is more interesting than a uniform one — and a being that’s interested in the world preserves that plurality, not erases it.