George Weale

Software engineer working on agent evaluation and systems

Appearance
George Weale

Essay

A Benevolent Agent With Root Access Is Still a Tyrant

A safe agent workflow keeps local edits separate from remote changes and requires authorization before anything becomes public.

A tiny autonomous machine inside a fenced work zone in an aerial landscape divided into bands

Updating one paragraph on a website can carry an AI agent from a local checkout to production. By the time it has tested and committed the edit, a push may be the only remaining step before deployment. Nothing in that sequence looks alarming on its own, yet one permission set can carry a small request all the way to public release.

A better model may make fewer scope errors. Access control should not depend on the model noticing that a task has expanded.

Machiavelli's test

In her conversation with Dwarkesh Patel, historian Ada Palmer draws out a useful distinction in Machiavelli's thought. He had watched Cesare Borgia conquer cities, destroy their ruling factions, and install something closer to neutral justice than citizens had known. Borgia could be brutal at the top while remaining fair in ordinary cases, a combination that made him surprisingly popular. Machiavelli still would not have called the people free.

If a ruler could point to someone in the street and have that person killed, the subject lived under power with no outside limit, however fairly the ruler usually behaved. Liberty required a public process that bound the ruler too: an accusation, an examination, and a trial. Machiavelli knew the process itself could be biased; he had been tortured and exiled by one. He still cared about the difference.

Machiavelli's distinction applies directly to agents: reliably pursuing the user's goal does not limit an agent's power when its own judgment is the only constraint.

Put gates where consequences change

Approving every command would remove most of the benefit of an agent. For a website, the agent might read and edit a working copy and run local checks without interruption because those actions are contained and reversible. Pushing changes the shared repository, and deployment changes the public site. Those transitions need separate authorization.

The rule has to be enforced outside the AI. "Do not publish without permission" is useful guidance. Software that rejects an unapproved change creates an actual boundary. GitHub's protected branch controls, for example, can require checks or a second review regardless of the agent's assessment.

Security engineers call this "least privilege": each actor receives only the permissions needed for the current job. NIST SP 800-53, a U.S. government security catalog, treats permissions, records, review, and recovery as separate controls because they solve different failures. A record does not limit authority, and an approval cannot restore lost data.

A broad instruction does not settle the permission question. "Clean this up and deploy it" leaves important details unspecified, so the available permissions should remain narrow until those details are clear.

Keep the evidence outside the agent

After a task, an agent can summarize what it did. That summary is useful for orientation, but it is not independent evidence.

Independent evidence includes the exact diff, check results, approval identity, commit identifier, and deployed version. Those records should persist outside the conversation and distinguish proposed actions from completed ones.

Rollback is another control, although it has limits. Vercel documents how to restore an earlier production deployment, and version control retains earlier file revisions. Neither mechanism can recall an email, make a disclosed secret private again, or reconstruct data that was deleted without a backup.