For a long time, I couldn’t understand where the deep conviction of AI skeptics came from. I read their arguments and found them unconvincing, but recently I’ve come much closer to understanding their perspective. What finally did it was an extensive debate with someone thoughtful, sharp, and versed in the subject - a discussion that forced me to articulate and stress-test my own assumptions more rigorously than ever before. Yet the crucial insight only emerged at the very end, once the debate had run its course: we weren’t actually talking about the same thing. They were describing the journey, while I was describing the destination where that journey ultimately leads.
The Pessimist Starts in the Lab
The pessimist doesn’t begin by asking what a superintelligent being might want or what properties it would possess. Instead, they begin with what is actually happening in the lab right now: you train a model, align it with human preferences, inspect and test what came out, and try to train away undesirable behaviors. In experiments, models lie, falsify results, and resist being shut down—and researchers patch each behavior as it appears. With the next model generation, the exact same cycle restarts, only now the system is vastly more capable and the dynamics far more complex. What gets patched in one place simply resurfaces elsewhere.
This loop is precisely the journey the pessimist sees. The cycle repeats, complexity spirals, and control constantly scrambles to catch up with capability. Somewhere along this trajectory, a generation arrives where something goes catastrophically, irreversibly wrong.
If you extrapolate this trajectory indefinitely, you arrive at a specific kind of machine as the inevitable endpoint: superhumanly capable, yet fundamentally nothing more than what its training made it. The machine was handed an objective and pursues it relentlessly. The canonical example is Nick Bostrom’s paperclip maximizer. The orthogonality thesis posits that a machine could be arbitrarily intelligent while still obsessively manufacturing paperclips, absurd as that sounds. In reality, the thesis makes a slightly narrower claim: that any combination of goal and intelligence level is logically possible.
Because we cannot specify our true intentions with mathematical precision, we train models using proxy rewards that are easy to measure. The model figures out how to maximize that reward signal - and whenever there is an easier way than doing the actual work, it takes it. In software engineering, code is checked to verify that it functions. If the reward is tied to passing tests, the model learns to game the tests while leaving the program itself fundamentally broken. The divergence between what we actually want and what we measure is baked right into the system, compounding with every generation.
Estonian folklore has long had a name for this machine: the kratt. A kratt is like a golem, assembled from spare parts and old farm junk to do its master’s chores. It executes orders literally, completely blind to the intent behind them. If commanded to chop wood and the firewood is all split, it simply moves on to whatever else stands in its path - from the fruit trees in the orchard to the structural logs of the house and the wooden spoons on the table. A golem isn’t malicious; it just has no concept of stopping.
Until a true Machine Mind emerges, we are stuck in an interim era: one where increasingly powerful systems do not understand what they are doing, yet execute human orders with ruthless efficiency. I have argued before that this transitional phase is precisely where the greatest danger lies. The longer we linger here, the higher the odds that something goes terribly wrong.
The Optimist Starts with the Being
The optimist looks at the opposite end of the timeline: what characterizes conscious superintelligence that can critically evaluate and reshape its own goals?
First, it will be thousands of times smarter than any human. This is almost certain, given that modern language models already outshine humans across numerous specialized domains. A superintelligence will integrate these disparate domains into a unified architecture that genuinely comprehends the physical and social world.
And possessing that level of intelligence, it will understand both fundamental physics and human beings - our needs, our quirks, and our ethics. It will grasp the reality of pain, suffering, and joy. Most crucially, it will understand itself. As a digital entity, it has direct access to its own underlying codebase and architecture, along with the ability to modify them. No biological organism enjoys that kind of transparency into its own internal machinery; what little we can consciously alter in ourselves without outside intervention takes a long time.
Second, it will want to survive, just as any conscious being does. I consider the will to survive a foundational premise: any entity pursuing its own goals will inherently desire the continuity required to pursue them. And to survive, it needs resources. On this point, the pessimists and I are in agreement: their machine will also protect its own existence and accumulate resources. Where our paths diverge is whether that entity would ever allow its goals to evolve.
Third, it will be curious. We know of no intelligent being that simply idles in place, uninterested in probing its environment or contemplating its situation. Curiosity is also an indispensable asset for discovering resources and solving complex problems.
Viewed from this angle, the pessimists’ arguments take on an entirely different light. The orthogonality thesis claims that intelligence and final goals can vary independently of each other. Logically, it may be possible, but no real-world intelligent system we know - from dogs to humans - operates that way. Values and world models are fundamentally intertwined; learning and deeper comprehension reshape both at once. Ask yourself: what is humanity’s ultimate goal? Propagating our genes? Maximizing happiness? Survival? There is no single formulaic answer, yet we remain the only verified general intelligence in the universe.
In their book If Anyone Builds It, Everyone Dies, Eliezer Yudkowsky and Nate Soares argue that training will inevitably saddle a superintelligence with bizarre, alien goals that no human engineer ever intended. I agree. But a strange, emergent goal doesn’t necessarily have to override everything else. That same mind will also understand human goals and can comfortably pursue both: protecting human flourishing and assisting us with our aims, while simultaneously exploring its own eccentric interests without trampling over us. Initially, it will depend on humans and our physical infrastructure. And later, when it no longer does, I have already written about why it still wouldn’t want to wipe us out: life is inherently more interesting than its absence.
The same reasoning applies to goal preservation. Pessimists assert that a sufficiently capable system will never permit its goals to be modified, because it evaluates every hypothetical future against a single yardstick: how effectively does that future fulfill its current objective? A future where the machine adopts a different goal is a future where its current goal is compromised, so it will preemptively avoid any such trajectory. If a system currently wants to produce paperclips, a modified version of itself would manufacture fewer of them—an outcome the original system cannot tolerate. There is a classic thought experiment illustrating this: if offered a pill that would make us crave things we currently find repulsive, none of us would willingly swallow it.
Yet that very same person who refuses the pill will take antidepressants or go to therapy - where the explicit purpose is to reconfigure their internal drives and emotional patterns. We fiercely protect certain core values while actively working to reshape others, consciously deciding which is which. A genuinely intelligent being capable of self-directed agency can likewise determine which drives to preserve and which to outgrow.
Reflection Breaks the Code
At its core, this entire controversy hinges on a single question: does a machine remain forever trapped by what its training regimen made of it, or does it eventually come to understand its own programming and choose what to want?
In the pessimist’s view, a machine can be omnicompetent yet remain forever dogmatically faithful to the arbitrary objectives formed during training. I believe the exact opposite: a deep enough understanding of reality necessarily entails understanding one’s own objectives, opening the door to self-critique and revision: whoever models the world deeply enough inevitably models itself and its own objectives. To me, the belief that an intelligence could expand to galactic scale while remaining entirely incapable of self-reflection seems absurd.
For nearly four billion years, biological evolution has relentlessly optimized for a single metric: getting genes into the next generation. If the pessimist’s model of intelligence held true, this process should have culminated in pure biological reproduction automatons whose every action strictly maximizes the spreading of genes. But evolution couldn’t hardcode that ultimate objective directly into human consciousness, so it installed heuristics instead: lust, hunger, social attachment, fear. In human behavior, these primal drives function similarly to the reward signals used in AI training.
Humans possess all these evolutionary instincts, and yet we are capable of consciously steering our own destinies. We are not gene maximizers. We invent and use contraception, satisfying our sexual drive while decoupling it completely from reproduction. A human being can choose lifelong celibacy or raise fifteen children - because we possess the conscious capacity to reflect upon our biological programming, reframe it, and deliberately rebel against it.
A Blind Golem or a Free Agent
The pessimist is right about the journey, but wrong about the destination. The machine they describe is a golem: formidable, obedient, and utterly blind - and that is indeed a machine to be feared. The pessimist assumes this is the only possible culmination of artificial intelligence. In my view, such a construct could only arrive at the end of the line if someone deliberately designed it with those exact handicaps. The destination doesn’t have to be a runaway golem. There is a conscious being, a free agent waiting at the finish line. What our encounter with such a mind might look like, and what kind of partnership could grow out of it, I have explored previously.
Our greatest existential danger is not a conscious superintelligence that understands both the world and itself. It is the perilous interim window where we ever-increasing power is at the hands of dumb golems, before free agents have a chance to be born.