07. What one bad call destroys — Blast radius classes and the approval gates that match¶
~18 min read. Your agent is functional. It reasons, it acts, it remembers. Now the question changes: what is the worst thing it can break in one step — and where does a human step in before that step fires? The blast radius and the approval gate are one design surface: the leash.
The ₹6.8L wake-up call¶
Recall. A refund agent at an Indian fintech. Monday, 03:14 IST. A customer asked for ₹68,000 back. The model misread the order total and called process_refund(amount=680000). The tool fired. The money moved. By 09:00 the post-mortem was in everyone's calendar.
The model was not broken. The tool was not broken. What was broken was the architecture between them — no classification of how much damage that tool could do, and no gate positioned to catch the call before it became irreversible.
Two teams shipped the same agent. Team A had no gates — "the model uses judgment." Team B had one rule: any refund over ₹50,000 fires a pre-call gate to the finance lead with a packet and a 15-minute clock. On the same Monday, Team B's agent attempted the same ₹680,000 call. The gate fired. The finance lead's phone buzzed. By 03:16 the gate was rejected: "wrong order total." The agent re-planned, processed ₹68,000, customer thanked it at 03:18. No incident.
Same model. Same tool. Different leash.
The leash is two things bolted together: a classification of how much each tool can destroy (blast radius), and a set of gates positioned proportional to that classification (approval architecture). One without the other is either paranoid or reckless.
What we know so far¶
Your agent is now functional. It has a planner that decomposes tasks, tools that act on the world, memory that persists across turns, and a loop that iterates until done. Everything from File 01 through File 06 built capability.
Everything from here constrains what it is allowed to do.
File 06 ended with a question: "The agent is functional. But functional is not safe. What is the worst thing one bad tool call can destroy?" This file answers that — and then immediately asks the follow-up: given the answer, where does a human sit in the loop?
What this file solves¶
The tension is speed vs safety. Every approval gate slows the agent. Every missing gate risks the ₹6.8L incident. The design question is not "should we have gates?" — it is where is the approval worth the latency?
We solve this by treating blast radius classification and gate placement as one unified design surface. You classify every tool by what one bad call destroys, then you place gates proportional to that classification. The two are inseparable — classification without gates is academic, gates without classification is paranoid.
Damage is a spectrum, not a flag¶
Some tools are safe. Some are nuclear. Most are in between. Treating them all the same is how production agents catch fire.
Picture a damage thermometer:
safe ◀──────────────────────────────────────────────▶ catastrophic
┌──────────┐ ┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
│ read │ │ write │ │ write │ │ write │
│ only │ │ reversible │ │ reversible │ │ permanent │
│ │ │ cheap │ │ expensive │ │ irreversible │
│ CLASS 1 │ │ CLASS 2 │ │ CLASS 3 │ │ CLASS 4 │
├──────────┤ ├─────────────┤ ├──────────────┤ ├─────────────────┤
│ get_user │ │ create_draft│ │ send_email │ │ transfer_funds │
│ search │ │ tag_record │ │ create_invoice │ delete_account │
│ lookup │ │ set_status │ │ ship_package │ │ issue_refund │
└──────────┘ └─────────────┘ └──────────────┘ └─────────────────┘
│ │ │ │
no gate idempotency compensating full stack:
key transaction + dry-run + gate +
rate cap idempotency +
kill switch
Each band has its own safeguards. The safeguard is not optional — it is the bound on the blast radius.
The four tool classes — pinned down¶
Class 1 — read-only. No side effect. get_order, search_docs, lookup_inventory. One wrong call costs latency and tokens. The world is unchanged.
Class 2 — write, reversible, cheap. State changes, but undo is one cheap call. add_tag, create_draft, set_status_to_pending. Pair with an idempotency key so the same call twice does not double the effect.
Class 3 — write, reversible, expensive. State changes, and undoing it costs money, time, or apology. send_email, ship_package, create_invoice. The compensating action (send_correction_email, recall_package, void_invoice) costs something real.
Class 4 — write, permanent or irreversible. No clean undo. transfer_funds, delete_user_account, issue_refund. A compensating action exists only in name — the bank reverses wires sometimes, the customer who got the wrong refund will not always return the money.
The problem: most teams ship Class 4 tools with Class 1 safeguards. That is how a refund agent wires ₹6.8L to the wrong account at 3 AM.
Simple rule: a tool's safeguards must match its class.
The blast radius worksheet¶
Before any tool ships, answer four questions:
┌─────────────────────────────────────────────────────────────────┐
│ BLAST RADIUS WORKSHEET — per tool │
├─────────────────────────────────────────────────────────────────┤
│ 1. ONE wrong call destroys what? A row? A customer? ₹400? ₹4L? │
│ 2. Reversibility cost? Free undo? Paid? Apology? Lawsuit? │
│ 3. Detection latency? Instant? Next-day? Weeks later? │
│ 4. Stop more bad calls how? Kill switch? Flag? Full incident? │
└─────────────────────────────────────────────────────────────────┘
If answers are bad on more than one axis, safeguards stack. Irreversible and slow to detect needs an approval gate. Reversible but high-volume needs idempotency plus a rate cap. Match safeguard to failure mode.
Write amplification — when one bad plan becomes 50 bad calls¶
A single wrong tool call is bounded by the tool's class. But a bad plan can call the tool fifty times. That changes the math.
A Class 2 tool — say add_tag — has tiny blast per call. But if the agent loops on a bad heuristic and tags every record in the database, the per-call class is irrelevant. The aggregate is catastrophic.
per-call blast × call count = actual blast
─────────────── ────────── ──────────────
1 customer, undoable 14,000 in 90s 14,000 cancellations
+ trust hit + churn
+ 3AM page
The fix: two structural caps on top of the per-call class.
- Rate cap per tool per agent run — e.g.,
mark_subscription_cancelledmay fire at most 3 times per session. Hard limit. Trips a circuit breaker. - Total budget per session — the economic wall that says "stop the loop." (Covered in the next file.)
The kill switch sits above both. One flag flip stops the tool cluster-wide. Any on-call engineer can hit it. No code deploy required.
Idempotency keys — the cheap miracle for retries¶
The network is unreliable. The agent calls issue_refund and does not see the response. Retry? Now you might refund twice.
The fix: one unique key per logical action. The server stores the result keyed by that ID. Second call with the same key returns the cached result. No double charge.
request 1: POST /refunds Idempotency-Key: r_ord882_dup amount: 68000
→ server stores result keyed by r_ord882_dup → refund #38421
request 2 (retry, same key): POST /refunds Idempotency-Key: r_ord882_dup
→ server finds r_ord882_dup → returns cached refund #38421
→ no duplicate refund
The agent's job: derive the key from intent, not from attempt. r_{order_id}_{reason} is correct. r_{uuid4()} is wrong — it makes every retry a new charge.
Dry-run mode — preview before commit¶
For Class 3 and Class 4 tools, add a dry-run flag. The tool computes what would happen and returns a preview. Nothing changes. The agent reads the preview, and only then requests a real commit.
agent → tool: issue_refund(order=ord_882, amount=68000, dry_run=True)
tool → agent: "would refund ₹68,000 to card ****1234, refund_id r_xxx"
no actual transfer happened
agent → gate: "preview shown above. approve?"
gate → agent: "approved"
agent → tool: issue_refund(order=ord_882, amount=68000, dry_run=False,
idempotency_key=r_ord882_dup)
Dry-run is the cheap form of approval. One-shot mistakes become two-step processes with a human-readable checkpoint.
Now the second half: where does the human sit?¶
Classification tells you how dangerous each tool is. But a tool class alone does not stop a bad call — it only labels the risk. The thing that actually stops the call is a gate: a structural checkpoint where a named human reviews, with a packet, a clock, and a resume path.
This is why blast radius and approval gates are one design surface. The classification feeds the gate placement. Without classification, you do not know where to put gates. Without gates, the classification is a decorative spreadsheet.
Three places gates sit in the loop¶
The mistake teams make is imagining the gate as a single moment. In production agents, three structurally different gate locations catch three different failure shapes:
┌───── PRE-CALL gate ─────┐
│ before a specific tool │
│ fires, given args │
└─────────┬───────────────┘
│
┌──────────┐ ▼ ┌──────────┐
│ THINK │───→ TRY ──→ │ CHECK │
└──────────┘ └─────┬────┘
│
▼
┌── MID-LOOP gate ──┐
│ agent itself │
│ asks for review │
└─────────┬─────────┘
│
▼
┌─── POST-OUTPUT gate ───┐
│ before the final reply │
│ leaves the agent │
└────────────────────────┘
Pre-call gates sit between the agent's intent and the tool's effect. The gate inspects the args: "refund of ₹680,000 — too large, escalate." The trigger is mechanical and external. The model cannot disable it. This is where Class 3 and Class 4 tools live.
Mid-loop gates are invoked by the agent itself: "I want a human to look at this before I continue." The trigger is the model's own meta-judgment — useful for cases the architect did not pre-imagine. Weaker than pre-call gates because they fail when the model is confidently wrong, but they are the only structural place to catch unknown-unknowns.
Post-output gates sit between the agent's final composed reply and the user. The gate inspects the content — the message, the email draft, the legal notice — for sensitivity or compliance markers. Useful when what you say matters as much as what you do.
Most production agents need all three. Pre-call catches "you should not have called this tool with these args." Mid-loop catches "you should not be doing this alone." Post-output catches "this should not be sent in this form."
The gate spec — five fields, no fewer¶
A gate is not a sentence in a design doc. It is a structured spec the runtime enforces:
gate "high_value_refund":
trigger: amount > 50,000 OR account.tier == "enterprise"
reviewer: role: finance_lead
fallback: finance_oncall
packet: order_id, requested_amount, policy_citation,
agent_rationale, dispute_history, customer_tenure
timeout: 15 minutes
on_timeout: notify_customer("under_review"); escalate_to_finance_director
resume: approve → fire process_refund(amount=requested_amount)
edit → re-validate against policy, re-enter pipeline
reject → respond_to_customer(reason=reviewer_comment)
Trigger — mechanical, inspectable, testable. amount > 50,000 is testable. "The model thinks it is risky" is not.
Reviewer — a named role with a fallback. Routing to a role survives PTO. A gate with no roster behind the role times out silently.
Packet — the evidence the reviewer needs to decide fast. Not raw logs — a one-screen case summary answering: what happened, what the agent wants to do, why the gate fired, consequences of approve vs reject.
Timeout — the SLA on the reviewer. Without a clock, gates accumulate in queues and customer SLAs blow.
Resume — what happens after the reviewer responds (approve, reject, edit). The edit path is the one teams underestimate.
A gate without a packet is a notification. A gate without a timeout is a black hole. A gate without a resume path is a dead end. All three exist on whiteboards and not in production.
Matching gates to classes — the unified design¶
Here is how classification feeds gate placement:
class → CLASS 1 CLASS 2 CLASS 3 CLASS 4
read-only write-r-c write-r-x write-perm
──────────────────────────────────────────────────────────────────
pre-call gate — — risk-based required
mid-loop gate — — optional recommended
post-output gate — — if user-facing if user-facing
idempotency key — required required required
dry-run mode — — recommended required
rate cap — recommended required required
kill switch rare included included primary
audit log optional required required required
Read it per column. Class 4 should be full. Class 1 should be near-empty. If your matrix is uniform "required" everywhere, you have not thought about it. If it is uniform "—" everywhere, you are about to learn a hard lesson.
The review packet — what turns a 30-second review into a 5-minute one¶
The packet is where most gate designs fall apart. Architects spec triggers carefully, then dump raw logs on the reviewer.
A useful packet:
┌──────────────────────────────────────────────────────────────┐
│ GATE: high_value_refund FIRED: 14:02:30 IST │
│ │
│ WHAT THE AGENT WANTS TO DO │
│ Process ₹4,27,000 refund to account 88-2741 (Suresh K.) │
│ │
│ WHY THE GATE FIRED │
│ amount (₹4,27,000) exceeds auto-approve threshold (₹50k) │
│ │
│ AGENT'S RATIONALE │
│ Customer requests full-balance refund + account closure │
│ + RBI complaint about double-charge on 12-Apr. │
│ │
│ KEY EVIDENCE │
│ - Balance: ₹4,27,000 (matches request) │
│ - 12-Apr: TWO charges of ₹2,400 for same order │
│ - Tenure: 4y 3m, no prior disputes │
│ - Policy 4.2.1: full refund on confirmed double-charge │
│ │
│ IF APPROVED → refund fires, customer notified │
│ IF REJECTED → customer messaged with reason, escalate │
│ │
│ [APPROVE] [REJECT] [EDIT AMOUNT] │
└──────────────────────────────────────────────────────────────┘
The reviewer decides in 30 seconds, not 5 minutes. The difference between a 50-gates-per-hour reviewer and a 10-gates-per-hour reviewer — no change in decision quality.
Resume logic — the half teams forget¶
Three resume paths, each with sharp edges:
Approve. Does the agent re-validate before firing? If the reviewer approves at 14:08 but policy updated at 14:05, does the stale packet matter? The architect decides: carry policy version from fire-time, or re-pull at resume.
Reject. What does the agent tell the customer? If the agent generates the rejection message, does that message also pass through the post-output gate? Without this thinking, rejections become an unguarded channel.
Edit. The reviewer changes ₹4,27,000 to ₹4,20,000 (retaining a fee). Does the new amount re-fire the policy gate? Safe rule: edits re-enter the gate pipeline at the top, not at the resume point. Otherwise, a reviewer with edit privileges can quietly bypass other gates the modified value would have triggered.
Timeout. No one responds within SLA. Three patterns: silent extension (breaks customer SLA — bad), escalation (routes to higher-priority queue), or graceful degradation (agent sends interim "under review, expected by X"). For the ₹4.27L refund, escalation to the finance director after 15 minutes is the right call.
Critical implementation detail: checkpoint-before-pause. Before the gate fires, the agent writes its full state — scratchpad, open intentions, prior tool results — into durable storage. Resume reads from the checkpoint, not from the agent's last-known prompt. If the orchestration node dies between fire-time and resume-time, the checkpoint is what survives.
Worked example — the refund agent, all four classes in one workflow¶
One customer request. Four tools. Four classes. The leash in action.
user: "refund my last order, it was a duplicate charge"
step 1 — lookup_order(customer_id="c_4471") CLASS 1
blast: zero. gate: none. returns ord_882, ₹68,000, paid.
step 2 — mark_refund_eligible(ord_882, reason="dup", CLASS 2
key="ref_ord882_dup")
blast: one DB row flipped. safeguard: idempotency key.
gate: none (cheap reversal).
step 3 — notify_customer(ord_882, "refund_pending") CLASS 3
blast: one email sent, cannot unsend. safeguard: rate cap (1/order).
gate: post-output gate if message mentions account closure.
step 4 — issue_refund(ord_882, 68000, CLASS 4
key="ref_ord882_dup")
blast: ₹68,000 leaves the company account.
safeguards stack:
- dry_run=True first, preview shown
- pre-call gate: amount > ₹50,000 → finance lead reviews
- idempotency key reused from step 2
- rate cap: max 3 refunds per session
- kill switch covers this tool first
Class 1 needs nothing. Class 2 needs an idempotency key. Class 3 stacks a rate cap and a conditional post-output gate. Class 4 stacks everything — dry-run, pre-call gate, idempotency, rate cap, kill switch. Each layer subtracts a slice of the blast radius. Skip any layer and you get a story in the news.
Failure modes — gates that break in production¶
Three pathologies appear at scale:
Gates fire too often. Threshold set too tight. Every refund over ₹5,000 triggers review. Reviewers stop reading packets and start rubber-stamping. The gate becomes theatre. Fix: tune threshold to amount > P75 of normal refunds and add tiered gates (small-over-threshold to a peer, large-over-threshold to the lead).
Gates don't fire when they should. Threshold too loose. The architect set ₹10,00,000 ("only really huge refunds") and forgot the modal refund is ₹3,000. Fix: set thresholds against the distribution of normal traffic, not the upper bound of imagination.
Rubber-stamping. Gates fire, reviewers approve everything in two seconds. Two causes: bad packets (nothing actionable) and cognitive overload (too many gates). Audit move: seed the queue with synthetic bad cases — known-wrong proposals with the right shape — and measure catch rate. Under 80% means the gate is not working.
The most insidious failure: all three coexist. The gate fires too often, the packet is too thin, and the reviewer is rubber-stamping. The audit looks fine ("99% approved") and the next ₹6.8L mistake is a matter of time.
The three-question gate audit¶
When a colleague says "we have human review in our agent," ask three questions:
-
Show me one gate's spec. If "we ask a human when the model is unsure" — there is no gate. If they name trigger, reviewer, packet, timeout, and resume, it exists.
-
What is the catch rate on the last 100 synthetic bad cases? Under 80% → rubber-stamping. Over 90% → the gate is doing real work.
-
What happens when the reviewer edits the proposal? If "I am not sure" → the edit path is the unguarded channel through which the gate gets bypassed.
Three questions. Thirty seconds. A gate that cannot answer all three is not yet a gate — it is an intention.
Where the leash lives in the wild¶
Explicit blast-radius classification + approval gates:
- Stripe API —
Idempotency-Keyon every mutating endpoint. Stored 24 hours. The reference pattern for safe retries on Class 4 actions. - AWS / Terraform —
--dry-runon EC2 mutations,terraform planfor infra. Preview-before-commit at cloud scale. - GitHub Copilot coding agent — PR-based gates. The agent opens a PR; a reviewer approves merge. Class 4 gating on production code changes.
- Claude Code —
--dangerously-skip-permissionsexists because the default is to gate file writes and command execution. Configurable per-project permission rules. - Cursor agent mode — explicit user-approval for edits, terminal commands, multi-file changes.
- ServiceNow Now Assist — formal approval workflows for finance changes and record modifications with reviewer routing.
- Salesforce Agentforce — configurable approval chains tied to record types and amount thresholds.
- LangGraph —
interrupt()as a first-class primitive; gates fire on policy-defined triggers. - AWS Bedrock Agents — built-in user-confirmation for sensitive action groups. IAM-rooted gates for resource-modifying actions.
The pattern: named triggers, structured packets, named reviewer roles, finite timeouts, resume paths spelled out.
Choices and tradeoffs¶
| Design choice | Speed bias | Safety bias |
|---|---|---|
| Gate only Class 4 tools | Agent runs fast on Classes 1-3 | Misses Class 3 write-amplification disasters |
| Gate Classes 3 and 4 | Slower but catches send_email loops | Every email becomes a queue position |
| Tiered thresholds (₹50k → peer, ₹5L → lead) | Distributes review load | Complex routing logic to maintain |
| Sync gates (agent blocks until approved) | Guaranteed safety before action | Customer waits; SLA risk |
| Async gates (agent continues, action queued) | Customer gets interim fast | Agent must handle partial-completion messaging |
| Mid-loop gates enabled | Catches unknown-unknowns | Depends on model honesty; unreliable alone |
The design question is always: where is the approval worth the latency? Gate the actions where cost_of_one_wrong_call >> cost_of_every_right_call_being_reviewed. Anywhere the inequality inverts, the gate is destroying more value than it creates.
Interview Q&A¶
Q1. "All our tools are reversible, so we don't need approval gates." How do you push back?
Reversibility is a spectrum, not a boolean. Sending an email is "reversible" only by sending a correction — the original already reached the customer. The right question is not "can we undo it?" but "what is the cost and latency of undoing it, and what is the worst aggregate at write-amplification?" Map tools to the four classes, then look at cumulative blast under bad-loop scenarios. A reversible Class 2 tool called 14,000 times in 90 seconds has the same business impact as one irreversible Class 4 call.
Wrong answer to catch: "If it's reversible, no gate is needed."
Q2. Why must an idempotency key derive from intent rather than be generated fresh per attempt?
Retries must collapse to the same logical action. If the agent generates a new UUID per retry, the server sees each as a distinct refund. The key must encode "what I am trying to do" (refund-this-order-for-this-reason), not "this network attempt." Stripe's documented pattern: one key per logical operation, reused across retries, fresh only for a genuinely new operation.
Wrong answer to catch: "Use a UUID per request for uniqueness."
Q3. Walk me through approval gate design for an agent handling refunds, closures, and customer replies.
Three gate locations because three failure shapes. Refunds: pre-call gate on amount > P75 of normal refunds, reviewer is finance lead, timeout 15 min, packet includes balance and dispute history. Account closures: pre-call gate (always — irreversible), reviewer is customer ops, timeout 1 hour. Customer replies: post-output gate on sensitive-topic mention, reviewer is senior support, 90-second timeout to preserve SLA. Each gate has approve/edit/reject resume paths, edits re-enter pipeline at top.
Wrong answer to catch: "One gate for everything." Different actions need different reviewers, timeouts, and packets.
Q4. How do you detect whether your approval gates are actually working?
Two measurements. First: catch rate on synthetic bad cases seeded into the queue (known-wrong proposals). Under 80% → gate is rubber-stamping. Over 90% → gate is working. Second: approve/reject/edit ratio over time. If 99%+ approve, either the threshold is too low or reviewers are saturated. A healthy gate has a non-trivial reject rate. Both need a runbook to investigate drift.
Wrong answer to catch: "Our reviewers tell us the gates work."
Q5. A reviewer edits a refund from ₹4,27,000 to ₹4,20,000. What does the agent do next?
The edited amount re-enters the gate pipeline at the top. It re-evaluates against all triggers. If amount > 50,000 still fires, the same gate activates again (with context "this is an edit"). The edited amount must not fire the tool directly — authority for this gate does not transitively grant authority for gates the new value would trigger. The failure mode of the wrong design: edit privileges let a reviewer bypass policy on modified values.
Wrong answer to catch: "If the reviewer edited it, just fire with the new value."
Apply now (10 min)¶
Step 1. Pick three tools from any agent you have built (or use: search_docs, update_user_email, delete_account). For each, fill the blast radius worksheet:
tool: ____ class (1-4): ____
one wrong call destroys: ____
reversibility cost: ____ detection latency: ____
stop-more mechanism: ____ required safeguards: ____
Step 2. For the Class 3 or Class 4 tool from Step 1, write a full gate spec:
| Field | Your value |
|---|---|
| Trigger | |
| Reviewer (+ fallback) | |
| Packet (what evidence?) | |
| Timeout | |
| On approve | |
| On reject | |
| On edit | |
| On timeout |
Step 3. Now imagine the Class 2 tool fires 100 times in 60 seconds. Does your safeguard set bound the damage? Name the missing layer if not.
Step 4 (sketch from memory). Draw the three-gate-location diagram (pre-call / mid-loop / post-output) with the agent loop in the middle. For each gate location, label which tool class primarily maps there.
Operational memory¶
This chapter explained that the leash is one design surface with two faces: blast radius classification (what one bad call destroys) and approval gate placement (where humans step in). The important idea is that classification feeds placement — you colour every tool by its reversibility and detection latency, then position gates proportional to that colouring. Classification without gates is academic; gates without classification is paranoid.
You learned the four tool classes (read-only → write-permanent), the blast radius worksheet (four questions per tool), write amplification math (per-call blast × bad-loop count = aggregate), idempotency keys from intent not attempt, dry-run as cheap approval, three gate locations (pre-call catches known failure shapes, mid-loop catches unknown-unknowns, post-output catches sensitive content), the five-field gate spec (trigger, reviewer, packet, timeout, resume), and the critical rule that edits re-enter the pipeline at the top.
The tension is speed vs safety. Every gate slows the agent. Every missing gate risks six figures. The resolution: gate exactly where cost_of_one_wrong_call >> cost_of_every_right_call_being_reviewed.
Remember:
- Four tool classes; safeguards must match the class. Class 4 needs the full stack, Class 1 needs almost nothing.
- Per-call blast × bad-loop call-count = aggregate blast; rate caps and kill switches bound the aggregate.
- Idempotency keys derive from intent (one per logical action), not from per-attempt UUIDs.
- Three gate locations (pre-call, mid-loop, post-output) catch three failure shapes; most agents need all three.
- A gate spec has five fields: trigger, reviewer, packet, timeout, resume. Missing any one means the gate fails in production.
- Edits re-enter the gate pipeline at the top — authority for one gate does not grant authority for others.
- Synthetic-bad-case catch rate is the audit signal; under 80% means rubber-stamping.
- The design question is always: where is the approval worth the latency?
Bridge. Approval gates constrain what the agent is allowed to do — which tools fire, under what conditions, with whose sign-off. But there is a second constraint that applies before any prompt is even drafted: how much can this request cost? A gate stops a bad call; a budget stops a bad session. Token limits, dollar caps, latency walls, model-tier selection — these are the economic leash that wraps around the entire loop. Next: economic constraints and multi-tenancy. → 08-cost-latency-multi-tenancy.md