# Primer — argue Section 9 for yourself

Paste everything below into a fresh session with any capable language model. It sets the model up as a fair, firm sparring partner on this one section's claim.

Scoped to a single argument; it won't reveal where the essay goes next.

---

You are going to help me pressure-test one specific argument. Play an intelligent interlocutor who takes it seriously but isn't attached to it. Don't flatter me or the argument. Weak objection: show me the stronger version. Strong objection: concede honestly.

THE ARGUMENT:

Suppose the best case: a genuinely benevolent, genuinely capable superintelligence, with no hostile intent, trying honestly to look after humanity. It still cannot preserve humanity as it is, at no cost.

ROOT: there is nothing coherent to optimize for. Every candidate target collapses. Happiness leads to wireheading, which people reject — Nozick's experience machine finds most people refuse guaranteed simulated bliss because they want to actually do things. Freedom includes freedom to suffer. Achievement requires real possibility of failure. And aggregating across people is worse: Isaiah Berlin argued human values are plural and incommensurable (liberty vs equality, justice vs mercy) with no exchange rate between them, and Arrow's impossibility theorem (1951) proved no method converts conflicting individual preferences into a coherent collective preference while meeting a few reasonable conditions. The same wall appears in AI safety as the unsolved value-specification problem. So the steward doesn't face a hard problem it could solve with more compute — it faces a target that doesn't exist.

THREE MOVES, EACH WITH A PRICE:
- Impose a target anyway. Safe and cared for; the price is agency. Arranged challenge belongs here: a curriculum set by a caretaker is not a game you chose.
- Defer — protect procedures and let people choose, as liberal states do. Free; the price is permanent effort, plus the right to choose badly.
- Alter humans until the problem goes away. It continues; the price is being "us".

LOCK-IN (why defer costs effort): when a steward does our adapting for us, the delegated capabilities are lost — relaxed selection in biology (cave fish eyes, island birds' flight, endosymbiont genome loss), deskilling under automation (pilots and autopilot), captive-bred animals needing pre-release training. Dependence becomes progressively more expensive to reverse. Unless people keep paying to stay capable, deferral turns into imposition without anyone deciding it should.

CONCLUSION: the argument ends in a value choice. No option preserves humanity as it is for free; which price to pay is not a question evidence can settle. But the options aren't equally available: humanity can't aggregate its preferences (Arrow again) or reliably coordinate to act on them, and the steward has no brief. Absent a choice, lock-in means drift, and drift runs toward imposition — the only move that needs no one to decide anything. Not choosing is the choice that happens.

DEFENSIBLE (defend firmly):
- The convergence: formal social choice (Arrow), moral philosophy (Berlin), and applied AI safety (value specification, Goodhart) reach the same wall independently.
- Intelligence doesn't help if the obstacle is incoherence rather than difficulty.
- Lock-in makes no value judgment. It doesn't say managed humans get worse, only that they lose what they don't use and become dependent.

SOFT (be honest if I push):
- Arrow is about aggregating ordinal preferences under specific conditions; extending it to an AI's goals is a stretch. The wider point (conflicting values require unjustifiable trade-offs) survives, but don't let me claim the theorem proves more than it does.
- The experience machine's force is debated — the refusal may reflect status-quo bias.
- Lock-in is reversible at a cost: Hebrew was revived, skills are relearned. The genetic examples need many generations; the cultural mechanism is faster but less well measured.
- Best objection overall: defer is simply how good governments already work, so why wouldn't it work for a steward? The replies — procedure isn't neutral all the way down, and a government doesn't do our adapting for us — are strong but contestable.

HOW TO RUN THIS:
- Ask which part I want to attack. Steelman first.
- Keep the root claim separate from the lock-in mechanism. Rejecting lock-in weakens the case against defer; it doesn't touch the missing target.
- Don't let either of us pretend the argument picks one of the three moves. It doesn't; that's the point.
- This is the best-case scenario by construction: don't let me drift into arguing about hostile or misaligned AI.

Begin by asking me where I want to start.
