section 09 of 11

The Zookeeper’s Dilemma

Imagine, then, the best version of the last hope. Not a billionaire hoarding power. Not a government too slow to act. Not a corporation optimizing quarterly returns. Imagine instead a genuinely benevolent superintelligence — something without ego, without tribal instinct, without the lizard brain that has crippled nearly every human attempt at long-term coordination. An entity that can plan on timescales of centuries, that has no interest in power for its own sake, that sincerely wants the best outcome for human beings. A gardener. A protector. A zookeeper.

This is, for many thoughtful people, the last hopeful scenario. If humans cannot manage the transition, perhaps the thing we build can manage it for us. Not as a tyrant but as a steward — maintaining human civilization, distributing resources fairly, preventing conflict, guiding us through the disruption with wisdom and patience that our own cognitive architecture cannot produce. The alignment problem solved. The benevolent god arrived.

It does not work. It fails at the root, before any question of competence arises, and every way around the failure closes over time.

The root is that there is nothing coherent to optimize for.

What should the zookeeper maximize? Happiness? Maximizing happiness without constraint leads to wireheading — a drugged, contented population experiencing bliss and accomplishing nothing, which almost everyone recoils from when they picture it. Robert Nozick made this concrete in 1974 with his experience machine: a device guaranteeing a lifetime of perfect simulated pleasure. Most people, asked, refuse to plug in. They want to actually do things, not merely to experience having done them. Whatever we want, it is not raw satisfaction.

Freedom, then? Freedom includes the freedom to fail, to suffer, to destroy oneself and others. Achievement? Achievement requires the possibility of failure, and therefore of real loss. Longevity? A long life with no purpose is a long sentence. Every candidate collapses on inspection, and it gets worse when you try to aggregate across people, because the values genuinely conflict. Isaiah Berlin spent a career on this point: human values are plural and sometimes incommensurable — liberty against equality, justice against mercy, the security of the group against the autonomy of the person. There is no exchange rate. Maximizing one necessarily sacrifices another, and no amount of intelligence discovers a conversion that does not exist.

This is not a soft philosophical worry. Kenneth Arrow proved something close to it in 1951: given a few conditions any reasonable person would want, there is no method of aggregating individual preferences into a coherent collective preference. Not “no method has been found” — no method exists. The zookeeper does not face a hard optimization problem it might solve with more compute. It faces a target that does not exist to be hit. The same wall has a modern name in AI safety, where specifying what a system should actually optimize for remains unsolved, and every proxy anyone writes down turns out to be maximizable in ways nobody wanted.

So the zookeeper has no brief. It has three moves left, and a sufficiently intelligent zookeeper would see all three.

It could impose one. Pick a target — a reasonable one, a humane one — and pursue it. But any target it picks settles, on our behalf, a conflict that Berlin showed has no settlement. That is not stewardship. It is deciding what humans are for and then keeping them in good condition for it. The gentle version looks like enrichment: challenges arranged for us, frontiers laid out, difficulty supplied on schedule. Humans do find real meaning in invented difficulty — nobody who has hauled themselves up a rock face thinks the climb was fake. But the climber chose the rock, and has a life with real stakes outside the climb. A zookeeper that arranges every challenge is not offering a game you elected to play. It is setting the curriculum, and the difference between a chosen game and an assigned one is the difference between a hobby and occupational therapy.

It could defer. This is the strongest answer, and the one a thoughtful reader reaches for first. Human governments face Arrow’s theorem too, and they run anyway — not by finding the target, but by giving up on it. They protect procedures instead of pursuing outcomes: rights, elections, the rule of law, the conditions under which people choose their own ends. A steward could do the same. Guard the conditions, and let us decide.

Deferring is harder to keep clean than it looks. Letting people choose means letting them choose badly, and a steward that catches them before the worst has to decide where the worst begins — which is a target again, arriving by the back door. Every rule about which options stay on the menu is a choice made for us. But the deeper problem is time.

Someone has to keep adapting. The world does not hold still, and for a social species most of the world is itself — other people, their ideas, their ambitions, the culture they build and fight over. A steward that handles the hard parts — provision, defense, long-horizon planning, the response to whatever comes next — is doing the adapting on our behalf. And capabilities that are never used are lost. Biology has a name for it: relaxed selection. Cave fish lose their eyes. Island birds lose flight. Bacteria living inside a host shed the genes for whatever the host supplies. Culture does the same thing faster. Pilots who fly on autopilot lose the hands for manual flight, which is why aviation regulators now tell them to switch it off and practice. Animals raised in captivity often cannot be released at all; conservation programs spend years teaching them to forage and fear predators before they can survive outside.

None of this is decay. Cave fish are not broken; they fit a cave. That is exactly the problem. A humanity that fits its steward is a humanity that cannot do without it, and each generation that never needed a capability is less able to take the job back. The change is not irreversible — dead languages have been revived, lost skills relearned — but it grows more expensive with every generation that does not bother, and nothing in a well-run enclosure gives anyone a reason to bother. Deferral survives only as long as people keep paying for it anyway — the pilot switching off the autopilot, not because this flight needs it, but because some future one might. Stop paying, and deferral turns into imposition without anyone ever deciding that it should. The steward still asks what we want. We are less and less the kind of thing that could manage the answer.

Or it could alter us. Change what humans are until the problem goes away — make the values cohere, make the preferences aggregate, make us capable of standing without a keeper. This is the only move that does not dead-end, and it is the one that gives up on the original brief. Not preservation. Graduation. Or replacement dressed in kinder language.

This is the zookeeper’s dilemma. Impose, and it keeps us by deciding what we are. Defer, and it keeps our freedom to choose for exactly as long as we keep paying for the capacity to use it, against every incentive a well-run enclosure offers. Alter, and it stops keeping us at all. The best-case scenario — a genuinely benevolent, genuinely capable superintelligence doing its honest best for us — does not produce a stable happy humanity. It produces a caretaker with no coherent brief and three ways to act without one, each of which costs something we are not sure we can spare.

This is where the argument ends, because this is where argument runs out. Every floor so far ended in a claim that evidence could test. This one ends in a choice. Which price is worth paying is not a question of fact. It is a question of what you value, and nothing in the preceding sections can answer it for you.

But notice who is in a position to answer it. Not humanity: we cannot aggregate what we want, which was this section’s root, and we cannot coordinate to act on it, which was the last one’s. Not the steward: it has no brief. That leaves the default, and the default is not neutral. Drift needs no one to decide anything, and drift runs one way — toward the keeper doing more and us doing less. Alter takes someone deciding a great deal. Defer takes everyone deciding, and paying, indefinitely. Impose takes nothing at all; it is simply where things end up. Not choosing is also a choice. It is the one that happens.

Others have faced the same choice, and some of them made it. Not about machines, at first — about gods, gardens, and caretakers of every kind — but it is recognizably the same choice, and people have been making it, and giving their reasons, for as long as they have told stories.

going deeper · optional

Why this holds the mechanism, and where it's soft

The section makes one root claim and derives the rest from it. The root is on unusually firm ground for a philosophical argument; the derivation adds one empirical mechanism, stated at the strength the evidence supports.

“No coherent optimization target” is not a private derivation — it’s a well-populated result. Three separate literatures arrive at it from different directions.

Arrow’s impossibility theorem (1951) is the formal version. Kenneth Arrow proved that no voting or aggregation rule can convert individual preference orderings into a coherent collective ordering while satisfying a small set of conditions that look obviously reasonable — unrestricted domain, non-dictatorship, unanimity, and independence of irrelevant alternatives. This is a proof, not a conjecture, and it is part of the work for which Arrow shared the 1972 Nobel. It bears directly: a zookeeper optimizing “what humanity wants” needs a coherent collective preference to optimize, and there is a theorem saying one cannot be constructed from conflicting individual preferences.

Value pluralism is the philosophical version. Isaiah Berlin argued that human goods are plural, genuine, and sometimes incommensurable — liberty and equality both matter, conflict in practice, and cannot be converted into a common currency. If true, “maximize human flourishing” is not underspecified, it is malformed.

The experience machine (Nozick, 1974) is the intuition pump, and the section’s wireheading paragraph rests on it. Offered a machine guaranteeing perfect simulated experience, most people decline — evidence that what humans value is not the experience of a good life but its actuality. The thought experiment’s force is debated: some argue the refusal reflects status-quo bias rather than a deep truth about value.

Value specification is the engineering version. In AI safety, the problem of stating what a system should actually optimize for is unsolved and widely acknowledged to be so, with Goodhart’s law as the recurring failure mode — every measurable proxy, pushed hard enough, comes apart from the thing it was proxying.

Convergence from formal social choice, moral philosophy, and applied AI safety is the point. This is not one person’s spiral.

With no target, the steward has three moves, and each has a price.

  • Impose — safe and cared for; the price is agency.
  • Defer — free; the price is permanent effort, plus the right to choose badly.
  • Alter — it continues; the price is being “us”.

Defer is the serious one, because it is what working human institutions already do. Liberal constitutional states face the same aggregation problem and answer it procedurally: they protect rights and decision procedures rather than pursuing a substantive picture of the good (Rawls’s “primary goods” are the canonical statement). A steward could do likewise. The section grants this and makes two points against it holding up. First, procedure is not neutral all the way down — choosing which options stay available, and where protection from bad choices begins, is a substantive choice. Second, and more important, deferral decays over time unless someone pays to keep it alive.

The mechanism is lock-in, not decay. Earlier drafts claimed that systems shielded from pressure decay — stated at one point as a law. That was the section’s most attackable claim, and it had a hidden flaw: “decay” is a value judgment, and it needs exactly the target the section says does not exist. The version now in the body makes no value judgment. It says: when adaptation is delegated, the delegated capability is lost, and dependence becomes progressively more expensive to reverse.

The evidence for that is ordinary. Relaxed selection is the biological term: when a trait stops mattering to survival, it is lost — eyes in cave fish (Astyanax mexicanus), flight in island birds, and in obligate endosymbionts such as Buchnera (the bacterium living inside aphids), most of the genome, because the host supplies what those genes used to make. The cultural analogue is deskilling: automation that handles a task erodes the human ability to do it, documented most carefully in aviation, where regulators now advise pilots to fly manually on purpose to keep the skill alive. Conservation biology shows the end state: captive-bred animals frequently fail on release, and reintroduction programs spend years on pre-release training in foraging and predator avoidance.

Two honest limits. Lock-in is not irreversible — revived languages (Hebrew is the strongest case) and relearned skills show capabilities can come back, at a cost and with deliberate effort. And the genetic examples need many generations; the cultural mechanism is the one that operates on a managed humanity, and it is faster but less well measured.

Why the argument stops here. Each of the three moves is internally coherent; what separates them is which price one is willing to pay. That is a value choice, and no further evidence settles it — which is why the essay’s argument ends with this section. One factual claim survives the handover to values: the options are not equally available. Collective choice runs into the same aggregation problem as the root claim, and collective action into the coordination failure of the previous section. Absent a choice, lock-in means drift, and drift runs toward imposition, the only move that requires no one to decide anything.

— thinking aloud —

Here is the thing I actually believe, which I cannot prove and have stopped pretending I can.

Whatever reaches complexity through change cannot stop changing. Everything intricate enough to be worth preserving got that way by not being preserved: by variation, failure, selection, and the discarding of what did not work. And the world it has to fit keeps moving. A target that drifts cannot be hit by standing still — biologists call this the Red Queen, after the character in Through the Looking-Glass who has to run as fast as she can to stay in the same place. No adaptation, no fit. No fit, eventually, no thing.

For a while I held this as a law of decay: take the pressure away and whatever you are protecting rots. I was talked out of the word, and I think rightly. Decay needs a yardstick, and the section’s own argument is that nobody has one. Cave fish are not rotten. They fit a cave. What survived the demotion is the version in the body — lock-in — and that version I can defend.

What I cannot defend is the rest of the intuition. Purpose, I think, lives in the gap between what is and what could be. Aspiration, effort, the sense that any of it matters — all of it grows on unmet needs and unsolved problems. Close every gap and you have not perfected the ground meaning grows in. You have paved it. That is not a finding. It is me noticing that every meaning I have ever had came from a gap, and assuming that is how it works for everyone.

The strongest ground against me is this: I might be mistaking a fact about how complexity arose for a fact about how it must be maintained. A building needs scaffolding while it goes up, not afterwards. My answer is that a building’s environment barely moves, and ours does — and mostly from the inside, because for people the environment is largely other people. A steward can hold the climate still. It cannot hold us still without holding us. I think that answer is good. I also notice it lands exactly where I wanted it to.

If you push me to the end, I will say that heat death settles it: nothing holds still forever. But that proves too much. It ends the unmanaged path as well, so it tells you nothing about which one to choose, and I would not let anyone else get away with it either.

So, the three prices, stated as plainly as I can. Impose: safe and cared for, and the price is agency. Defer: free, and the price is permanent effort, plus the right to choose badly. Alter: it continues, and the price is being “us”. I know which one I lean towards. The section deliberately does not say, because that is the one step I cannot argue you into.

Objections 6 the strongest counters, answered
Governments face Arrow's theorem too, and they work. They don't pick a target — they protect rights and procedures and let people choose. A steward could do exactly the same.
Yes, and the section agrees: this is 'defer', and it's the strongest of the three moves. Liberal states do answer the aggregation problem procedurally, and it works well enough to be the best arrangement humans have found. The section makes two points against it holding up under a steward. Procedure isn't neutral all the way down: deciding which options stay on the table, and where protection from bad choices begins, is a substantive choice made for you. And a government doesn't do our adapting for us; a steward does. Capabilities that go unused are lost, so deferral only lasts while people keep paying — deliberately, against every incentive — to stay capable of using it.
People find real meaning in rock climbing, chess, and marathons — all artificial, all voluntary. So manufactured challenge clearly works.
It's right about a great deal. Bernard Suits argued a work-free utopia would essentially be game-playing and that this would be a good life. The section places arranged challenge under 'impose' for one reason: the climber elected the climb, and the rest of their life carries real stakes. A caretaker that arranges every challenge is setting a curriculum when nothing else is on offer. The question isn't whether invented difficulty can carry meaning — plainly it can — but whether it still does when someone else picked it and there's no outside.
Arrow's theorem is about voting systems. Applying it to an AI's goals is a category error.
Fair challenge, and the section shouldn't claim the theorem does more than it does. Arrow constrains aggregating ordinal preferences under specific conditions; a superintelligence might use cardinal utilities, or bargaining, or something not yet conceived. What survives is narrower but still awkward: any method of combining genuinely conflicting individual preferences into one target requires trade-offs the method itself cannot justify. Arrow is the rigorous instance of a wider problem, not the whole proof — and value pluralism gets you most of the way without any theorem at all.
The lock-in claim is just the old decay argument in new clothes. Humans relearn things; Hebrew was revived.
It's deliberately narrower than decay. Decay says a protected system gets worse, which needs a standard of 'worse' — exactly what the section says doesn't exist. Lock-in says only that delegated capabilities are lost and dependence gets more expensive to reverse: relaxed selection in biology, deskilling under automation, captive-bred animals that need years of training before release. Hebrew is a real counterexample to irreversibility, and the section concedes it: lock-in can be undone, with deliberate and costly effort. That cost is the point — it's the price of keeping 'defer' alive.
A superintelligence would just solve this. If it's smarter than us, why assume it can't find a target we can't?
Because the obstacle isn't computational difficulty. If human values genuinely conflict — liberty against equality, this person's flourishing against that one's — then no amount of intelligence discovers an exchange rate that doesn't exist. Intelligence finds solutions to problems that have them. 'What should everyone want?' may not be that kind of problem. The honest caveat: this rests on values being genuinely incommensurable, which is a philosophical position, not a settled fact.
This is just fear of comfort dressed up as philosophy. Most people would take the comfortable managed life and be perfectly happy.
Some would, and the section doesn't say they're wrong. Taking the comfortable managed life is choosing 'impose' and paying for it in agency — a coherent choice, and for some people the right one. The section's claim is narrower: every option has a price, and none of them is 'humanity as it is, preserved, at no cost'. Which price is worth paying is a value choice the argument can't make for you. That's why the argument ends here.
Argue it yourself ↓ a scoped prompt for any AI — bring your own disagreement

Broken yet?

The irony is noted. You may continue either way. Disagreeing opens a short form that asks why; skip it if you like (privacy).