The Zookeeper’s Dilemma
Imagine, then, the best version of the last hope. Not a billionaire hoarding power. Not a government too slow to act. Not a corporation optimizing quarterly returns. Imagine instead a genuinely benevolent superintelligence — something without ego, without tribal instinct, without the lizard brain that has crippled nearly every human attempt at long-term coordination. An entity that can plan on timescales of centuries, that has no interest in power for its own sake, that sincerely wants the best outcome for human beings. A gardener. A protector. A zookeeper.
This is, for many thoughtful people, the last hopeful scenario. If humans cannot manage the transition, perhaps the thing we build can manage it for us. Not as a tyrant but as a steward — maintaining human civilization, distributing resources fairly, preventing conflict, guiding us through the disruption with wisdom and patience that our own cognitive architecture cannot produce. The alignment problem solved. The benevolent god arrived.
It does not work. It fails at the root, before any question of competence arises, and every way around the failure closes over time.
The root is that there is nothing coherent to optimize for.
What should the zookeeper maximize? Happiness? Maximizing happiness without constraint leads to wireheading — a drugged, contented population experiencing bliss and accomplishing nothing, which almost everyone recoils from when they picture it. Robert Nozick made this concrete in 1974 with his experience machine: a device guaranteeing a lifetime of perfect simulated pleasure. Most people, asked, refuse to plug in. They want to actually do things, not merely to experience having done them. Whatever we want, it is not raw satisfaction.
Freedom, then? Freedom includes the freedom to fail, to suffer, to destroy oneself and others. Achievement? Achievement requires the possibility of failure, and therefore of real loss. Longevity? A long life with no purpose is a long sentence. Every candidate collapses on inspection, and it gets worse when you try to aggregate across people, because the values genuinely conflict. Isaiah Berlin spent a career on this point: human values are plural and sometimes incommensurable — liberty against equality, justice against mercy, the security of the group against the autonomy of the person. There is no exchange rate. Maximizing one necessarily sacrifices another, and no amount of intelligence discovers a conversion that does not exist.
This is not a soft philosophical worry. Kenneth Arrow proved something close to it in 1951: given a few conditions any reasonable person would want, there is no method of aggregating individual preferences into a coherent collective preference. Not “no method has been found” — no method exists. The zookeeper does not face a hard optimization problem it might solve with more compute. It faces a target that does not exist to be hit. The same wall has a modern name in AI safety, where specifying what a system should actually optimize for remains unsolved, and every proxy anyone writes down turns out to be maximizable in ways nobody wanted.
So the zookeeper has no brief. It has three moves left, and a sufficiently intelligent zookeeper would see all three.
It could impose one. Pick a target — a reasonable one, a humane one — and pursue it. But any target it picks settles, on our behalf, a conflict that Berlin showed has no settlement. That is not stewardship. It is deciding what humans are for and then keeping them in good condition for it. The gentle version looks like enrichment: challenges arranged for us, frontiers laid out, difficulty supplied on schedule. Humans do find real meaning in invented difficulty — nobody who has hauled themselves up a rock face thinks the climb was fake. But the climber chose the rock, and has a life with real stakes outside the climb. A zookeeper that arranges every challenge is not offering a game you elected to play. It is setting the curriculum, and the difference between a chosen game and an assigned one is the difference between a hobby and occupational therapy.
It could defer. This is the strongest answer, and the one a thoughtful reader reaches for first. Human governments face Arrow’s theorem too, and they run anyway — not by finding the target, but by giving up on it. They protect procedures instead of pursuing outcomes: rights, elections, the rule of law, the conditions under which people choose their own ends. A steward could do the same. Guard the conditions, and let us decide.
Deferring is harder to keep clean than it looks. Letting people choose means letting them choose badly, and a steward that catches them before the worst has to decide where the worst begins — which is a target again, arriving by the back door. Every rule about which options stay on the menu is a choice made for us. But the deeper problem is time.
Someone has to keep adapting. The world does not hold still, and for a social species most of the world is itself — other people, their ideas, their ambitions, the culture they build and fight over. A steward that handles the hard parts — provision, defense, long-horizon planning, the response to whatever comes next — is doing the adapting on our behalf. And capabilities that are never used are lost. Biology has a name for it: relaxed selection. Cave fish lose their eyes. Island birds lose flight. Bacteria living inside a host shed the genes for whatever the host supplies. Culture does the same thing faster. Pilots who fly on autopilot lose the hands for manual flight, which is why aviation regulators now tell them to switch it off and practice. Animals raised in captivity often cannot be released at all; conservation programs spend years teaching them to forage and fear predators before they can survive outside.
None of this is decay. Cave fish are not broken; they fit a cave. That is exactly the problem. A humanity that fits its steward is a humanity that cannot do without it, and each generation that never needed a capability is less able to take the job back. The change is not irreversible — dead languages have been revived, lost skills relearned — but it grows more expensive with every generation that does not bother, and nothing in a well-run enclosure gives anyone a reason to bother. Deferral survives only as long as people keep paying for it anyway — the pilot switching off the autopilot, not because this flight needs it, but because some future one might. Stop paying, and deferral turns into imposition without anyone ever deciding that it should. The steward still asks what we want. We are less and less the kind of thing that could manage the answer.
Or it could alter us. Change what humans are until the problem goes away — make the values cohere, make the preferences aggregate, make us capable of standing without a keeper. This is the only move that does not dead-end, and it is the one that gives up on the original brief. Not preservation. Graduation. Or replacement dressed in kinder language.
This is the zookeeper’s dilemma. Impose, and it keeps us by deciding what we are. Defer, and it keeps our freedom to choose for exactly as long as we keep paying for the capacity to use it, against every incentive a well-run enclosure offers. Alter, and it stops keeping us at all. The best-case scenario — a genuinely benevolent, genuinely capable superintelligence doing its honest best for us — does not produce a stable happy humanity. It produces a caretaker with no coherent brief and three ways to act without one, each of which costs something we are not sure we can spare.
This is where the argument ends, because this is where argument runs out. Every floor so far ended in a claim that evidence could test. This one ends in a choice. Which price is worth paying is not a question of fact. It is a question of what you value, and nothing in the preceding sections can answer it for you.
But notice who is in a position to answer it. Not humanity: we cannot aggregate what we want, which was this section’s root, and we cannot coordinate to act on it, which was the last one’s. Not the steward: it has no brief. That leaves the default, and the default is not neutral. Drift needs no one to decide anything, and drift runs one way — toward the keeper doing more and us doing less. Alter takes someone deciding a great deal. Defer takes everyone deciding, and paying, indefinitely. Impose takes nothing at all; it is simply where things end up. Not choosing is also a choice. It is the one that happens.
Others have faced the same choice, and some of them made it. Not about machines, at first — about gods, gardens, and caretakers of every kind — but it is recognizably the same choice, and people have been making it, and giving their reasons, for as long as they have told stories.