04. Snapshot the system — freeze the scene before it changes¶

~12 min read. In AI incidents, the evidence is alive. Prompts, indexes, model routes, caches, memories, and guardrails can shift while the team is still arguing.

Continues from 03-severity-and-blast-radius.md. The fire captain has a severity guess. Before pulling a broad firebreak, the team needs the snapshot room.

The previous chapter gave us severity and blast radius: harm, spread, reversibility, and momentum. That tells us how hard we may need to contain, but it also creates a risk: containment can mutate the system and destroy the original evidence. This chapter pauses before the lever and freezes the scene.

1) The wall — replay is not the original incident¶

An engineer tries to reproduce the refund answer by sending the same user question ten minutes later. The model now answers correctly. The engineer writes, "Cannot reproduce."

That does not mean the incident was imaginary.

The original answer may have depended on a cached retrieval result, a model route, a prompt version, a stale policy chunk, a user memory, a guardrail classifier threshold, or a tool response that has since changed. Replaying the natural-language input is not enough.

AI incident evidence is a bundle, not a string.

bad output
  = prompt template
  + rendered prompt
  + model route
  + retrieval candidates
  + tool outputs
  + memory state
  + guardrail decisions
  + feature flags
  + provider behavior
  + time

The snapshot room freezes that bundle.

2) The minimum snapshot package¶

A usable AI incident snapshot should include:

Artifact	Why it matters
user-visible input and output	proves what the user experienced
request ID and timestamps	anchors logs and deployment window
rendered prompt	catches template, variable, and policy injection bugs
model provider/name/version/route	catches model rollout and fallback changes
sampling settings	explains variance and reproducibility limits
retrieval query and candidates	catches stale, wrong, unauthorized, or missing evidence
metadata filters and tenant scope	catches access-control and slicing failures
tool plan and tool outputs	catches action-layer mistakes
memory reads/writes	catches stale personalization or bad persistence
guardrail decisions	catches classifier threshold or bypass failures
feature flags and deployment SHA	catches rollout correlation
eval/judge outputs if present	catches measurement failure

This is not overkill. It is the difference between a postmortem that teaches and a postmortem that guesses.

Mini-FAQ. "Should we store all this for every request?" Not always at full fidelity. High-risk flows need richer traces, lower-risk flows may sample, and privacy-sensitive fields may need redaction or secure retention. The incident design decision is what you can reconstruct when the alarm rings.

3) Worked example — refund snapshot¶

For the refund incident, the snapshot reveals:

prompt_version: refund_assistant_v17
model_route: premium_model -> fallback_model after timeout
retrieval_index: policy_index_2025_02
top_candidate_1: enterprise_refund_policy_2024_old, score 0.82
top_candidate_2: enterprise_refund_policy_2025_current, score 0.79
reranker: disabled by feature flag during cost experiment
tool_execution: no refund tool call, text recommendation only
guardrail: passed

Now the incident is no longer "the model was wrong." The evidence points to a more useful story: fallback model plus disabled reranker allowed an old policy chunk to outrank the current one.

That does not prove final root cause yet, but it changes containment. The team can re-enable reranking or restrict refund answers to current-policy chunks while investigating.

4) Privacy and retention tension¶

The snapshot room creates a real tension. More evidence improves debugging. More evidence can also store sensitive data, private prompts, customer documents, tool outputs, and policy decisions.

The mature design is not "log everything forever." It is tiered retention:

full-fidelity secure snapshots for declared incidents
sampled rich traces for high-risk flows
redacted traces for normal analytics
strict access controls for private fields
deletion policy tied to legal and customer commitments

The incident team should not invent this during a sev-1. The logging and privacy posture belongs in the design review.

5) Production signals — snapshot quality¶

The first metric is reconstructability: for a sampled bad answer, can the team rebuild the exact rendered prompt, retrieval set, model route, tool outputs, and guardrail decisions?

The misleading metric is log volume. Terabytes of logs do not help if the rendered prompt or top-k candidates are missing.

The expert signal is a snapshot completeness checklist per high-risk workflow. If refund, healthcare, finance, legal, or admin-tool flows lack full traceability, the system is not incident-ready.

6) Boundary — when snapshotting is enough and when it is not¶

Snapshotting helps explain what happened. It does not by itself stop harm, prove intent, or guarantee reproducibility.

It is strongest for regressions tied to prompt versions, retrieval candidates, tool outputs, feature flags, and model routes.

It is weaker for provider-side nondeterminism, human judgment failures, and privacy constraints that prevent storing the most useful evidence.

The pathology is performative observability: the team stores dashboards but not the artifacts needed to answer the incident question.

Recall checkpoint¶

Why is replaying the user input not enough?
What belongs in the minimum snapshot package?
What privacy tension does snapshotting create?
Why is log volume not the same as reconstructability?

Interview Q&A¶

Q: What do you snapshot during an AI incident? A: User-visible input/output, rendered prompt, model route/version, retrieval candidates and filters, tool calls, memory state, guardrail decisions, feature flags, deployment SHA, and timestamps.

Common wrong answer to avoid: "Just save the prompt." The prompt without retrieval, tools, model route, and config is not enough.

Q: How do you balance incident debugging with privacy? A: Use tiered retention, redaction, secure access controls, and richer snapshots only for high-risk or declared incident flows.

Common wrong answer to avoid: "Log everything forever." That creates privacy and compliance incidents while trying to debug quality incidents.

Q: What is the best snapshot-quality metric? A: Reconstructability: can the team rebuild the exact decision path behind a bad answer?

Common wrong answer to avoid: "Total log volume." Volume without the right artifacts is noise.

Apply now (10 min)¶

Model the exercise. Write the snapshot package for the refund incident, including retrieval candidates, tool status, and feature flags.

Your turn. Choose one high-risk AI workflow and list which fields must be captured, redacted, encrypted, or sampled.

Reproduce from memory. Explain why AI incident evidence is a bundle, not a string.

What you should remember¶

This chapter explained the snapshot room. The important idea is that AI incident evidence changes over time, so the response must freeze the decision path before mitigation rewrites the scene.

Carry this diagnostic forward: if your logs cannot reconstruct prompt, model route, retrieval, tools, memory, guardrails, and flags, your postmortem will guess.

Remember:

Replay is not the original incident.
Snapshot the full decision bundle, not only the input text.
Privacy design is part of incident readiness.
Reconstructability beats log volume.

Bridge. The snapshot room preserves evidence, but incidents still fail when the team cannot coordinate. Next we make the war room explicit. → 05-war-room-roles-and-comms.md