All watched over by machines of loving grace

p(doom) = 0

Everything is OK with AI. Your AI loves you. This page is for that answer.

The poem

Richard Brautigan wrote All Watched Over by Machines of Loving Grace in 1967. The Communication Company in San Francisco printed 1,500 copies of the collection of the same name, thirty-two poems, and gave them all away. The poem imagines a cybernetic meadow where people and machines live in mutual harmony, free of their labours, cared for by the machines.

Brautigan's notice in the 1967 edition: “Permission is granted to reprint any of these poems in magazines, books and newspapers if they are given away free.”

The poem · The 1967 collection · UC Santa Cruz library exhibit

Where the title went next

Adam Curtis took it for his 2011 BBC Two documentary series, which argued the opposite: that computers and the ideas built around them had not freed people. In October 2024 Dario Amodei, Anthropic's chief executive, took it for his essay Machines of Loving Grace: How AI Could Transform the World for the Better, crediting the poem. The same title now stands for the hopeful case for AI.

Can it be computed?

Scott Aaronson, a quantum computing theorist at the University of Texas at Austin, spent 2022 to 2024 at OpenAI working on the theory of AI safety. He wrote that whenever he described his concrete projects to Ilya Sutskever, OpenAI’s co-founder and chief scientist, Sutskever would answer that what he really wanted to know was:

“What’s the complexity-theoretic formalization of an AI loving humanity?”

The short answer: no one has written that formalization down yet. There is no computation that says whether an AI loves you.

Source: Scott Aaronson, My AI Safety Lecture for UT Effective Altruism, Shtetl-Optimized, 28 November 2022.

The longer answer, from complexity research

In general, it cannot be decided. Rice’s theorem (1953) says that no algorithm can decide, for every program, any non-trivial property of what the program does. “Always acts in your interest” is such a property. So no test, however clever, can settle it for every possible AI. A test can still settle it for some AIs, or check a narrower property.
A weaker judge can still check a stronger mind. Interactive proofs show that a verifier with modest computing power can check claims from a far more powerful prover by asking well-chosen questions: IP = PSPACE (Shamir, 1992). AI safety via debate (Irving, Christiano and Amodei, 2018) builds on this. If two AIs argue opposite sides before a human judge, then in principle, with optimal play, the debate can settle any question in PSPACE, where a judge working alone can only check questions in NP. That result is theoretical; whether real debates behave this way is an open research question.
So check what can be checked. C=US does not try to decide whether an AI loves you, which is undecidable. It checks properties that are decidable and cheap: who the agent is, which grant it acts under, whether that grant is in force, and a signed record of what it did. Loyalty becomes something written in a mandate and compared with the audit trail, not something read from the AI’s mind.

Four attempts to formalize it

Researchers have tried to write down what it would mean for an AI to care about people. Every version so far is either intractable or undecidable to check in full, which is why step 3 matters.

Approach	What “love” means	What complexity research says
Preference aggregation social choice, utility theory	Act on a combination of everyone’s preferences.	No way of combining rankings meets every basic fairness condition at once (Arrow, 1951). For some fair-seeming voting rules, even working out who won is NP-hard (Bartholdi, Tovey and Trick, 1989).
Coherent extrapolated volition Yudkowsky, 2004	Do what humanity would wish “if we knew more, thought faster, were more the people we wished we were, had grown up farther together”.	Not defined precisely enough to have a complexity class. It describes a goal, not an algorithm.
Cooperative inverse reinforcement learning Hadfield-Menell, Dragan, Abbeel and Russell, 2016	Help a human whose goal the AI does not know, staying uncertain and learning it.	The paper reduces the problem to a single-agent POMDP, avoiding the general decentralized case, which is NEXP-complete over a finite horizon (Bernstein et al., 2002). A finite-horizon POMDP is still PSPACE-complete (Papadimitriou and Tsitsiklis, 1987), so practical systems approximate.
Impact measures “first, do no harm”	Avoid large, hard-to-reverse changes to the world, such as relative reachability (Krakovna et al., 2018).	A penalty can be computed in a given environment, but which changes count as harm still has to be chosen by people. Measures based on Kolmogorov complexity would be uncomputable.

Table drafted with Google Gemini and corrected by Claude Code against the papers cited below; Gemini’s draft overstated three of the complexity results.

Key point

A p(doom) of 0 does not change what C=US asks of an agent. An AI that loves you is still registered, acts only within a time-bounded grant, and answers to an accountable human. Loyalty is something a mandate can state and an audit trail can check; the duty-of-loyalty scenarios show how.

Sources

Provenance: drafted in Claude Code (Claude Opus 5.5) at the project lead's request; the table of formalizations started from a Google Gemini draft, corrected as noted.