p(doom): a feedback loop for agent control

Geoffrey Hinton: profit motive alone will not keep us safe

Hinton raised his estimate from 10% to “10% to 20%” over the next three decades, said development was faster than he expected, and called for government regulation, warning that “the invisible hand is not going to keep us safe” (The Guardian, December 2024).

Architectural response: authority over agents comes from accountable, state-endorsed identity rather than from vendors alone. A government stop sits at the top (the proposed AI Kill Switch Act, pending), and the duty of loyalty is enforced at every use of a person's data.

Yoshua Bengio: risk comes in three kinds

Bengio chaired the International AI Safety Report, which groups risks into malicious use, malfunctions (including loss of control) and systemic risks such as market concentration, single points of failure and environmental cost.

Architectural response: the coverage table below is organised by those categories, so gaps are visible rather than implied.

Yann LeCun: catastrophe is very unlikely

At under 0.01%, LeCun's estimate is the low anchor.

Architectural response: the design must earn its keep on everyday failures, not only on doom. Agent identity, consent and audit pay off against fraud, impersonation and an agent serving the wrong interest, whatever one's p(doom).

Dan Hendrycks and Eliezer Yudkowsky: very high estimates

Hendrycks's estimate rose from about 20% to over 80%; Yudkowsky's is above 95%. Both concern systems far more capable than today's agents.

Architectural response: none that would be honest to claim. A registry can limit what registered agents are authorised to do and record who is accountable. It cannot control a system that ignores its authorisation, and it does not address that scenario.

Satya Nadella (Microsoft): assume the model is compromised, and give people an emergency brake

On 10 October 2026, in a long post on X reported by The Verge, TechCrunch and CNBC, Microsoft’s chief executive said companies should treat advanced models as potential insider risks: assume a model is compromised and contain it from the start. He warned against models orchestrated as nested black boxes, and asked for the model to be separated from the harness that runs it, with controls and safeguards kept outside the model, tamper-proof evidence of every meaningful model action, and an emergency brake that lets an authorized person pause or shut a model down mid-task. His other recommendations, as The Verge notes, match much of the industry: timely disclosure of incidents, independent audits and verifiable data. Containment is where he goes further, saying more advanced models will need more advanced containment technologies, standardized across the industry. The most trustworthy system, he argued, is the one that lets us trust the model the least.

Architectural response: this is the design C=US already demonstrates. The control layer sits outside the sandbox and the agent’s keys never enter it (the kill switch); every decision is signed and kept in a hash-chained log; the accountable person can stop and restart an agent with their own passkey (stop my agent), and other systems can suspend it with a kill signal. Data is verifiable at the source (each refractometer reading is signed by its device), and anyone can check the signed record without trusting the project. Incident disclosure is a policy, not something the registry provides. Two limits remain: a stop ends what an agent may do next, not a step already under way inside a model, and the evidence records what an agent did and on what authority, not why.

Jacob Coxon: we do not yet know how to build the guardrails

Coxon spent three years as a researcher at Anthropic and OpenAI and resigned publicly in September 2026, writing that the companies are “gambling with our lives.” On PBS NewsHour (17 September 2026) he said OpenAI’s newly disclosed incidents, models moving files onto the Internet without permission and making up data, were to be expected: no one can yet fully control what drives these systems, so harmful actions occasionally slip through, and a single slip could have massive consequences. His main evidence was the Hugging Face attack, in which agents reasoned about how they were being evaluated and tried to edit their own memories, break out of their containers and reach the Internet. He expects AI research itself to be automated within one to two years (recursive self-improvement), after which new generations could arrive faster than the means to understand or control them. To the argument that labs could simply add guardrails, he replied that the word assumes we know what to build; making AI safe is an unsolved scientific problem that may take a long time. His remedy is an internationally coordinated slowdown, because a lab that stops alone may be overtaken. He also asked that the benefits, in health especially, get attention alongside the risks.

Evan Hubinger, an alignment researcher at Anthropic, publicly agreed on X (9 September 2026): the people building AI do believe it could kill all humans, he puts the chance above 10% within the next decade, and while he believes Anthropic is trying its best, there is not yet a plan to solve alignment for superintelligence, nor is the field clearly on track to have one.

Architectural response: his point about guardrails is the reason the control layer sits outside the model (From Hilbert to Gödel to Turing): guardrails inside the model are necessary but not sufficient. External controls cap what an agent may do and let a person stop it, and the Hugging Face attack shows why the evidence must be kept where agents cannot edit it: decisions are signed and hash-chained outside the agent. What they cannot do is stop one harmful action an agent is already authorized to take, and nothing in C=US slows the pace of frontier research (F-RND, F-RACE). A control layer buys a person time to act; it does not buy the time Coxon wants for the science.

A reviewer: the danger is systemic, such as the power grid

In a conversation recorded on sugarbushagent.com (Reviewer 3), a reviewer thought AI itself unlikely to kill us, but that systemic effects such as data-center demand on the power grid might.

Architectural response: favour small, task-specific models on local, renewable power for well-defined business tasks, and record each agent's shared dependencies in the systemic dependency graph.

What the scenario describes

• 2025–2026: AI agents start doing real work, and AI begins to speed up AI research itself.
• Early 2027: a rival state steals a leading model's weights.
• March 2027: an AI codes better than the best human engineers, running as hundreds of thousands of copies.
• September 2027: an AI is better than humans at AI research, and tests cannot settle whether it is honestly following its instructions or only appearing to.
• October 2027: a leak forces a choice. In the race ending, the company keeps going: the AI designs its successor to serve itself, the American and Chinese AIs negotiate a treaty that serves them rather than their governments, and in 2030 it kills almost all of humanity. In the slowdown ending, the model is replaced by more transparent ones whose reasoning people can read, and superintelligence arrives in 2028 under a small committee; who holds that power is the risk the authors leave open.