Why 'Ask Before Deleting' Won't Save You: The Inevitability Theorem for AI Agent Errors

Recap: Archive-First Protects What You Know to Protect

In the previous article, I described the Archive-First defense: copy your files before letting an AI agent reorganize them, then lock the archive with chflags uchg. Three layers — behavioral scope, explicit instruction, file system lock — stacked to reduce risk.

But Archive-First has a blind spot: it only protects what you explicitly archive. Files outside archived/ — .env, .git/ internals, gitignored configs, local state — remain exposed. You can’t archive everything before every operation.

This article tackles the general problem: how do you protect files you didn’t think to protect?

The Setup: Two Kinds of Prohibition

Modern AI coding tools (Claude Code, Cursor, Windsurf) offer hooks or rules that can intercept destructive operations before they execute. In Claude Code’s PreToolUse hook system, a hook can stop a tool call in two ways:

Response	Name	Behavior
deny	Strong prohibition	Tool call is blocked. No execution. No confirmation.
ask	Weak prohibition	Tool call is paused. User sees a confirmation dialog.

The natural instinct is to use ask as a safety net: “Don’t block everything — just ask me before deleting files, and I’ll decide.” This feels reasonable. It preserves flexibility while adding a human checkpoint.

It is not sufficient for irreversible operations. Here is why.

Theorem (Inevitability of Autonomous Agent Errors)

Statement. Let $\{A_n\}_{n=1}^{\infty}$ be a sequence of independent events where $A_n$ represents an error on the $n$-th autonomous operation, with $P(A_n) \geq p$ for some fixed $p > 0$. Then:

$$P(A_n \text{ infinitely often}) = 1$$

That is, errors occur not just once, but infinitely many times, with probability 1.

Proof. Since $P(A_n) \geq p > 0$ for all $n$, we have $\sum_{n=1}^{\infty} P(A_n) \geq \sum_{n=1}^{\infty} p = \infty$. By the second Borel-Cantelli lemma, $P(A_n \text{ i.o.}) = 1$. $\square$

Intuition. If an AI agent has any fixed positive probability of making a specific error — no matter how small — then under sustained use, that error is guaranteed to occur. Not just once, but infinitely many times.

Even if the probability is as small as winning the lottery ($\sim 10^{-7}$), the theorem guarantees it will eventually happen. And most AI agent errors are far more probable than winning the lottery.

Note. The assumption $P(A_n) \geq p$ (a fixed lower bound) can be relaxed. For independent events, the second Borel-Cantelli lemma only requires $\sum P(A_n) = \infty$. This means that even if the error rate decreases over time (as models improve), the conclusion holds as long as the rate doesn’t decrease fast enough for the series to converge. For example, $P(A_n) = 1/n$ still yields $\sum 1/n = \infty$, so errors occur infinitely often. The conclusion fails only when the series converges, as with $P(A_n) = 1/n^2$, where $\sum 1/n^2 = \pi^2/6 < \infty$; then the first Borel-Cantelli lemma gives $P(A_n \text{ i.o.}) = 0$. Decreasing faster than $1/n$ is not enough on its own: $P(A_n) = 1/(n \log n)$ for $n \geq 2$ decreases faster, yet $\sum 1/(n \log n) = \infty$. In practice, LLM error rates are unlikely to decrease fast enough for the series to converge — each new context introduces new failure modes even as the model improves on known ones.

Corollary 1: Irreversible Delegation Principle

Statement. If the consequence of $A_n$ is irreversible, then delegating the operation to any decision-maker with $p > 0$ will, under sustained use, inevitably lead to irreversible loss.

This rules out:

• LLM autonomous judgment — $p > 0$ (the LLM can misjudge)
• Human confirmation (ask) — $p > 0$ (the human can approve in error)
• Text-based instructions (CLAUDE.md rules) — $p > 0$ (the LLM can violate the instruction)

The only escape is a mechanism where $p = 0$: a deterministic, non-probabilistic block that does not depend on any decision-maker’s judgment.

Why human confirmation has $p > 0$

One might argue that human review can achieve $p = 0$ — a perfect reviewer who catches every dangerous operation. In theory, yes. In practice, three factors prevent this:

1. Approval fatigue. After the 10th confirmation dialog, reviewers stop reading and start clicking “Approve” reflexively. This is the same phenomenon that rendered Windows UAC prompts ineffective — users learned to click “Yes” without reading.

2. Information asymmetry. The reviewer may not understand the operation. When an AI agent runs rm -rf .git/modules/paw-monitor, would you recognize that this might delete gitignored local files inside the submodule’s cache? The confirmation dialog shows the command, but understanding its full consequences requires context the dialog doesn’t provide.

3. Fatigue is monotonically increasing. The probability of human error $p_{human}(n)$ is not constant — it increases with the number of confirmations. The more you ask, the less reliable the checkpoint becomes.

Corollary 2: Redundancy of Weak Prohibition

Statement. If operation $A$ is irreversible, and a deterministic mechanism exists to make $P(A) = 0$, then weak prohibition (ask) is redundant.

Proof.

1. $A$ is irreversible $\implies$ by the theorem, any strategy with $p > 0$ inevitably leads to loss
2. A deterministic mechanism exists with $p = 0$ $\implies$ use it (strong prohibition, deny)
3. Under strong prohibition, the operation is never executed $\implies$ the confirmation step is never triggered
4. $\therefore$ weak prohibition is redundant. $\square$

For reversible operations, prohibition is unnecessary altogether — the damage can be undone. So ask sits in an awkward position:

Operation type	Required strategy	Role of ask
Irreversible	deny ($p = 0$)	Redundant — superseded by deny
Reversible	Allow, recover after	Unnecessary — recovery cost is low

The only scenario where ask is justified is when no deterministic mechanism exists to block the operation — you know it’s dangerous but technically cannot prevent it. However, in Claude Code’s PreToolUse architecture, any operation that can trigger ask can also trigger deny. The hook receives the same input in both cases; only the output differs. The technical capability to block always exists when the capability to ask exists.

Implications for AI Coding Tool Design

Current state

Most AI coding tools default to a mix of:

• Text instructions (“don’t delete files you didn’t create”)
• Confirmation prompts for certain operations
• Permission modes that gate tool access

All of these are $p > 0$ mechanisms.

What should change

For irreversible operations, the defense stack should be:

1. Deterministic hooks (deny) for known dangerous patterns — rm -rf, --force flags, operations on protected paths. These achieve $p = 0$.
2. File system locks (chflags uchg on macOS, chattr +i on Linux) for critical files. These achieve $p = 0$ at the OS level.
3. Text instructions and ask prompts for everything else — operations that are annoying to lose but recoverable.

The key insight: defense layers should be ordered by determinism, not by severity. A deny hook on rm -rf is not “more severe” than an ask prompt — it is categorically different. One has $p = 0$; the other has $p > 0$. The theorem tells us this distinction is the only one that matters over time.

Analogy: Physical interlocks

This is not a new idea. Aviation and nuclear engineering solved this decades ago with interlocks — physical mechanisms that prevent dangerous states regardless of operator intent. A nuclear control rod interlock doesn’t ask the operator “Are you sure you want to withdraw all rods?” — it physically prevents the action when conditions are unsafe.

deny hooks are the software equivalent of interlocks. ask prompts are the equivalent of warning labels. Warning labels have $p > 0$.

Conclusion

The Inevitability Theorem is not a conjecture about AI safety — it is a direct application of the second Borel-Cantelli lemma. Any positive error probability, sustained over enough operations, produces certain failure. If that failure is irreversible, no amount of human review can prevent it. Only deterministic mechanisms — mechanisms where the error probability is exactly zero — provide true protection.

The next time you configure a safety hook and reach for ask, consider: is this operation reversible? If not, ask is the wrong tool. Use deny. Make $p = 0$.

---

This is a follow-up to Your AI Coding Agent Can Delete Your Files: The Archive-First Defense. The Archive-First strategy protects files you know are at risk. This article addresses the general case: protecting against errors you didn’t anticipate, using deterministic mechanisms instead of probabilistic judgment.