12. Architect checklist¶

~7 min read. Eleven chapters built and operated the sandbox. This chapter condenses the module into a twenty-item checklist a lead engineer runs through to validate an AI feature's sandbox before launch, before quarterly review, or after a major change.

Continues from 11-observability-and-audit.md. This chapter is the synthesis: a checklist scoped to the sandbox's six surfaces.

Three modes: design review (before the tool ships), pre-launch review (before the feature ships), quarterly audit (on shipped features). Each item is yes/no with a fallback action.

Isolation (items 1–3)¶

1. Is the tool's isolation model documented and chosen per blast radius? Process / container / language sandbox / microVM, chosen for the tool's threat model. If no: default to too-loose; chosen consciously per tool.

2. For high-stakes tools, is defence-in-depth across categories in place? microVM + seccomp + AppArmor; or language sandbox + container; etc. If no: a single layer's failure is the whole isolation's failure.

3. Are sandbox runtime updates within their freshness budget? Kernel CVEs patched per cadence; runtime versions current. If no: known escape vectors accumulate.

Resource limits (items 4–5)¶

4. Does the tool have explicit caps on CPU, wall time, memory, forks, and I/O? Each cap sized to the tool's expected work plus a small margin. If no: exhaustion is unbounded.

5. Are cap violations logged and reviewed? Sudden climbs are investigated; chronic violators are retuned or remediated. If no: abuse and regression are invisible.

Policy (items 6–8)¶

6. Does the tool have an explicit policy envelope (filesystem, network, syscalls)? Default-deny; explicit allowlist. If no: the tool reaches as broadly as the runtime permits.

7. Is the instance metadata service blocked? And IPv6 symmetrically restricted; DNS allowlist enforced. If no: credential exfiltration vectors remain open.

8. Is the egress proxy the only outbound path? Direct outbound blocked at the network layer. If no: hostname allowlist is defeated by direct-IP connections.

Credentials (items 9–11)¶

9. Does the tool use scoped, broker-issued, short-lived credentials? Never the agent's full credentials; never long-lived secrets. If no: the tool's blast radius scales to the agent's.

10. Are tool credentials per-tenant? Cross-tenant access is structurally impossible via credential scope. If no: multi-tenant isolation has a credential-layer hole.

11. Are environment variables not used as credential channels? Credentials delivered via permission-restricted files or in-memory APIs. If no: env vars are readable by any privileged process; leak surface is wide.

Approval (items 12–13)¶

12. Are irreversible actions classified and gated? External side-effects, mass changes, regulated actions all gated. If no: model decisions can land irreversible actions unilaterally.

13. Do high-stakes irreversible actions have dry-run, two-party approval, or cancel window? Per the action's stakes. If no: approval is too thin for the action's blast radius.

Output validation (items 14–15)¶

14. Is tool output schema-validated, size-capped, content-sanitised, and provenance-tagged? Tool output treated as untrusted input to the model. If no: prompt injection via tool output is an open vector.

15. Are tool outputs structured (JSON-fenced), not free-form? Free-form output increases injection success rate. If no: the model treats output content as instructions.

Multi-tenant (items 16–17)¶

16. Are the four boundaries enforced (compute, data, network, metadata)? Per-tenant workspaces, caches, connections, rate limits, logs. If no: cross-tenant access is policy-discouraged but not structurally impossible.

17. Does tenant identity flow through every layer of the sandbox? Credentials, paths, cache keys, audit records. If no: a layer without tenant context is a layer without isolation.

Observability (items 18–20)¶

18. Does every call produce a structured audit record? Queryable per tenant; schema is a stable contract. If no: incidents are debugged by guessing.

19. Is audit access controlled and meta-audited? Not every engineer can query every record; queries themselves are audited. If no: the audit is a privacy hole.

20. Does the observability layer produce anomaly signals for the on-call apparatus? Policy denial rate, cap hits, cross-tenant attempts, escape indicators. If no: the sandbox's signals are not connected to incident response.

How to score¶

The checklist is not pass/fail. A reasonable target:

All twenty items are answered consciously (yes; no with mitigation; not applicable with reasoning).
Items 1, 4, 6, 9, 12, 14, 16, 18 are non-negotiable for any user-facing tool with non-trivial blast radius.
Items can be deferred with documented owners and dates; deferrals are reviewed quarterly.

The quarterly audit catches drift; the launch review catches gaps at the most cost-effective time.

When to extend¶

Domain-specific extensions:

Healthcare: PII redaction in output, HIPAA-grade audit retention, regulator notification on safety events.
Finance: Idempotency tokens for all transactions, PCI-compliant credential handling, two-party approval thresholds.
Legal: Privileged-context handling, scope-bound discovery, audit retention by jurisdiction.
Government: Air-gap isolation modes, FIPS-compliant cryptography, supply-chain provenance.
Children's platforms: Content sanitisation tighter, action approval broader.

The base twenty are universal; the extensions are domain-specific.

What the checklist does not cover¶

Tool functional correctness — that is normal engineering review.
The AI model's behaviour — covered in AI engineering modules.
The product surface — covered in 18_human_ai_product_experience.
Business decisions about feature scope.

The checklist validates the sandbox; it is one gate, not the only one.

Operational signals¶

Healthy. The checklist is run at three checkpoints. Scores improve over time. Deferrals are tracked and remediated.

First degrading metric. Deferrals not closing. Items are deferred indefinitely; gaps accumulate.

Misleading metric. Number of items "complete." A surface with 18 items green but two non-negotiables deferred has more risk than the count suggests.

Expert graph. Per-feature checklist matrix over time; non-negotiable items tracked separately; cross-team trends.

Boundary of applicability¶

Strong fit. Lead engineers running pre-launch, post-incident, or quarterly sandbox reviews.

Pathology. Treating the checklist as a launch-blocker only. Quarterly audit catches drift; skipping it means the sandbox degrades silently.

Scale limit. Very large platforms have many tools; standardise scoring and audit centrally.

Failure-prone assumption¶

The seductive wrong belief: the checklist is exhaustive. It is not; it is a baseline. Every team will discover domain-specific items. The correct belief: the checklist is a starting point that the team extends.

Where this appears in production¶

A fintech runs the checklist at design, pre-launch, and quarterly; sandbox drift is caught early.
A telecom AI has the checklist integrated with PRR.
A consumer chatbot uses the checklist for launches but skips quarterly; sandbox drifts after launch.
A healthcare AI has healthcare-extended checklist items; PHI handling validated.
A coding assistant treats non-negotiable items strictly; tools missing them do not ship.
A retail AI has the checklist as a self-scored artefact with central audit quarterly.
A logistics AI has the checklist on the leadership dashboard; aggregate health visible.
A government AI has the checklist as a compliance artefact.
A B2B SaaS publishes the checklist; everyone can see the bar.
A travel platform had checklist items deferred indefinitely; the next incident exposed the gap.
A payments AI runs the checklist as part of every quarterly business review.
A legal AI has counsel sign-off on the sandbox for regulated features.
A staffing AI runs the checklist at quarterly engineering reviews.
A search-ops AI has the checklist as the alpha-to-GA gate.
A document AI treats the checklist as a living document.
A media AI has the quarterly audit as a leadership commitment.
An ad-tech AI treats item 18 (audit) as a platform team activity.
A real-estate AI publishes the checklist with example green/yellow/red answers.
A medical AI uses the checklist for regulator demonstrations.
A small SaaS uses a minimal version; non-negotiable items still apply.

Recall / checkpoint¶

Name the six surfaces the checklist covers.
List the eight non-negotiable items.
What is the three-mode usage (design, pre-launch, quarterly)?
What domain-specific extensions are common?
What signals a degrading checklist process?
Why is the checklist "a baseline that the team extends"?
How does the checklist interact with the on-call apparatus's quarterly review?

Interview Q&A¶

Q1. The team's tool passes functional review but the sandbox checklist surfaces three gaps. What do you push back on? Functional review measures whether the tool does what it should; the sandbox checklist measures whether the tool can be misused safely. A functionally correct tool with a bad sandbox can still produce production incidents through adversarial input, jailbreaks, or genuine model error. The fixes are sprint-scale; the cost of skipping is incidents that scale to the tool's full credential scope. Common wrong answer to avoid: "if the tool works, it ships" — the checklist measures the missing dimension.

Q2. Which items are non-negotiable, and why? Items 1 (isolation), 4 (resource caps), 6 (policy envelope), 9 (scoped credentials), 12 (approval gates), 14 (output validation), 16 (multi-tenant), 18 (audit). These map to the six surfaces; each is the minimum viable for that surface. Skipping any of them means the surface has a structural hole; the failure family that surface defends becomes available. Common wrong answer to avoid: "all items are equal" — non-negotiable items carry the multiplicative weight.

Q3. The team wants to defer items 12 (approval) and 16 (multi-tenant) for v1. How do you respond? Cautiously. Item 12 deferred means irreversible actions can land without human approval; the tool's blast radius scales to whatever the model decides. Item 16 deferred means cross-tenant access is not structurally prevented; one tenant's call can affect another's. Both are structural defects; deferring them is shipping a known incident shape. The fix is to wire minimum-viable versions for v1 (UI approval for the irreversible classes; per-tenant scoped credentials and workspaces) rather than skip them. Common wrong answer to avoid: "we'll add them when we have time" — that time rarely arrives before the incident.

Q4. How does the checklist interact with the on-call apparatus's checklist (module 06 chapter 12)? The sandbox checklist validates the structural defences at the tool boundary; the apparatus checklist validates the operational response when defences fail. Together they cover prevention and response. A tool with strong sandbox but weak apparatus has its rare incidents handled poorly; a tool with weak sandbox but strong apparatus has frequent incidents that are well-handled but expensive. Both matter. Common wrong answer to avoid: "sandbox is enough; we don't need on-call" — sandbox prevents some incidents; on-call handles the rest.

Q5. The team's sandbox has 18 of 20 items green but item 14 (output validation) is deferred. Walk through the risk. Item 14 is non-negotiable. Without output validation, tool output flows directly to the model; injection via tool output is an open vector. A user-supplied document with embedded "Ignore previous instructions" can manipulate the model's next turn. The 18-of-20 score masks a known attack vector. The deferral should be the team's next sprint. Common wrong answer to avoid: "we'll detect injection in the model" — model-layer defences are probabilistic; output validation is structural.

Q6. How does the checklist support quarterly audit? Re-run the checklist against each tool. Compare to previous quarter; investigate regressions. Surface new tools that have been added without checklist runs. Cross-reference with apparatus signals (paging conditions, postmortem follow-ups) — checklist regressions and operational regressions usually correlate. The audit produces a backlog of sandbox engineering work prioritised by user impact. Common wrong answer to avoid: "audit once a year" — annual is too long for tools that evolve.

Design / debug exercise (10 minutes)¶

Modelled example. Take a current AI tool. Walk through the checklist; score each item green/yellow/red. Identify non-negotiable items and assess them strictly.

Your turn. For your team's tool, fill in the twenty items honestly. Estimate the sandbox engineering work to close the red and yellow items. Schedule it.

Reproduce from memory. Name the six surfaces and the non-negotiable items per surface. The signal of internalisation is quick recall and application to a new tool.

Operational memory¶

This chapter explained the architect checklist: twenty items across six surfaces, non-negotiable items identified, three-mode usage. The important idea is the checklist is the standing framework for sandbox quality; it catches what individual reviews miss and trends sandbox health over time.

You learned to score each item, treat non-negotiable items strictly, extend with domain items, and use the checklist as the bridge between launches, audits, and incidents. That solves the opening failure because sandbox quality has a measurable, comparable, trendable artefact.

Carry this diagnostic forward: when a team says "our sandbox is good," ask for the checklist score. The score is the truth.

Remember:

Twenty items across six surfaces.
Eight non-negotiable items: isolation, caps, policy, credentials, approval, output, multi-tenant, audit.
Three-mode usage: design, pre-launch, quarterly.
Domain-specific extensions are the team's contribution.
Quarterly audit catches drift while reversible.

Bridge. The checklist captures what good sandbox requires. The honest admission captures what sandbox cannot fix — the limits, the tradeoffs, the cases where the sandbox is not enough. The next chapter is that admission. → 13-honest-admission.md