George Weale

Software engineer · agent quality and efficiency · ML systems

George Weale

Essay

A Benevolent Agent With Root Access Is Still a Tyrant

A helpful AI still needs narrow permissions, approval before public actions, a record of what it did, and a way back.

A tiny autonomous machine inside a fenced work zone in an aerial landscape divided into bands

An AI agent is an AI that can use tools and take actions, and "root access" gives it the master key to a computer, allowing it to change almost anything. Even updating one paragraph on a website crosses several boundaries. The agent reads the folder containing the site's code, edits a file, checks that the site still works, records a save point, sends the change to the shared copy, and finally makes it public.

Each step is reasonable on its own, yet the sequence is not harmless, because reading has become writing, writing has become publishing, and a local suggestion has become a public fact. Nothing malicious needs to happen; the agent only has to misunderstand the scope once.

We usually treat this as a question of how smart the AI is, assuming that a smarter agent will infer the request and a careful one will notice when the task has expanded. Better models certainly help. They do not explain why one fallible process should be able to carry an idea all the way from a prompt to a public website. Helpfulness is a motive, not a security boundary.

Machiavelli's test

Historian Ada Palmer describes a useful distinction in her conversation with Dwarkesh Patel. Machiavelli had watched Cesare Borgia conquer cities, destroy their ruling factions, and then install something closer to neutral justice than citizens had known; Borgia could be brutal at the top while remaining fair in ordinary cases, a combination that made him surprisingly popular. Machiavelli still would not have called the people free.

If a ruler could point to someone in the street and have that person killed, the subject lived under power with no outside limit, regardless of how fairly the ruler usually behaved. Liberty came from a public process that the ruler also had to use, including an accusation, an examination, and a trial. That process might itself be biased, as Machiavelli knew after being tortured and exiled by one, but he still cared about the difference.

A folder of code is not a republic, and a broken website is not political violence; the analogy concerns the shape of power. "This actor usually makes good choices" is a claim about character, while "this actor is unable to cross certain lines alone" is a property of the system. An agent with root access may be polite, reliable, and pointed at the user's goal, but if its own judgment is the only constraint, the power is still arbitrary.

Put gates where consequences change

Stopping for approval after every command would turn the agent into autocomplete with paperwork, so a better workflow gives it room to operate inside a boundary. For a website, the agent might read and edit its own working copy and check the result without interruption, because those actions stay local and can usually be undone. Sending the change to everyone else crosses one line, while making it public crosses another; those moments deserve separate permission.

Security engineers call this "least privilege," meaning that each actor receives only the permissions needed for the current job. A U.S. government security catalog, NIST SP 800-53, separates permissions, records, review, and recovery because each solves a different failure: a record does not limit authority, just as an approval cannot restore lost data.

The rule has to live outside the AI. "Do not publish without permission" is useful guidance, but software that physically rejects an unapproved change creates a real boundary. GitHub's protected branch controls, for example, can require checks or a second review even when the agent is convinced its work is ready.

Broad instructions do not deserve broad authority, because a request such as "clean this up and deploy it" leaves plenty of room for interpretation. When the language is ambiguous, the available permissions should narrow rather than widen.

Keep the evidence outside the agent

After a task, an agent can provide a clean account of what it did, which is useful for orientation but still amounts to a story written by the same agent whose actions we want to check.

The evidence lives elsewhere, in the exact change, the checks that ran, the identity that approved it, and the version that reached the public. That record should survive the conversation and clearly distinguish a proposed action from one that actually happened.

Being able to go back matters too, although its power has limits. Vercel documents how to restore an earlier public version of a site, and the code history keeps earlier versions of the files, yet neither can make someone forget an email, unpublish a leaked secret, or reconstruct destroyed data.

The workflow I want should be easy to describe without mentioning the agent's personality. It can read here, write there, run these commands, and ask for additional permission at a clearly marked boundary. A durable record shows what happened, and a tested mechanism restores the last known good state. If the only explanation is "it knows what I want," then the system does not have governance. It has a benevolent ruler with root access.