09. Escape vectors and defenses¶

~8 min read. The sandbox exists. The remaining question is how it is escaped. The discipline is to know the common vectors, harden against them, and monitor for the patterns that signal an escape in progress.

Continues from 08-output-validation-and-sanitisation.md. Recurring concepts in bold: escape vector, kernel CVE, container escape, TOCTOU, side channel, hardening cadence, escape monitoring.

Sandboxes are not unbreakable. The discipline is to know how they break and to design depth that survives a single layer's failure.

The common vectors¶

Vector	What it is	Where it lands
Kernel CVE	Vulnerability in the shared kernel	Container or process isolation
Container escape	Misconfigured capabilities, leaked devices, broken seccomp	Container isolation
Filesystem traversal	Path traversal past misconfigured chroot or mount	Filesystem policy
Network bypass	DNS, IPv6, side-channel routing	Network policy
Credential exfiltration	Reading env vars, metadata service, leaked tokens	Credential layer
Side channel	Cache, timing, hardware sharing reveals data	Multi-tenant isolation
Supply chain	Compromised library or base image	Build pipeline
TOCTOU	Time-of-check vs. time-of-use race	Policy enforcement

Each vector has a different shape and a different defence.

Kernel CVEs¶

The shared kernel is a single point of failure for container isolation. A kernel CVE that allows privilege escalation or container escape is a security incident the moment it is published.

The defence:

Rapid patching. Critical kernel CVEs patched within days, not weeks.
Reduced kernel surface. gVisor (user-mode kernel) or microVMs (separate kernel) reduce the host kernel surface visible to tools.
Defence-in-depth at the syscall layer. seccomp profiles deny syscalls that are common in escape vectors (unshare, keyctl, bpf, userfaultfd).
Monitor known indicators. Tools attempting to exploit common CVEs leave traces in syscall logs.

A team running containers on an unpatched kernel during a published CVE has a known incident waiting.

Container escape¶

Container escapes have happened repeatedly in production. Common patterns:

Misconfigured capabilities. Containers granted CAP_SYS_ADMIN or CAP_NET_RAW have higher escape surface.
Leaked devices. /dev access (especially /dev/kvm, /dev/mem) leaks abilities to the host.
Mounted Docker socket. A container with /var/run/docker.sock mounted can spawn privileged containers.
Path traversal in volume mounts. Symlinks in mounted volumes pointing outside.

The defence is configuration discipline: drop all capabilities by default, opt in to the minimum needed. No Docker socket inside containers. Volume mounts canonicalised. Read-only root filesystem.

Filesystem traversal¶

Even with chroot or container mounts, path traversal can escape if:

The application accepts user-supplied paths without canonicalisation.
Symlinks in the workspace point outside the workspace.
The chroot is incomplete (some special files like /proc/self still accessible).

Defence: canonicalise all paths before checking; resolve symlinks before allowlist check; use proper sandbox primitives (namespaces, microVM filesystems) rather than chroot for non-trivial isolation.

Network bypass¶

Common network policy bypasses:

DNS exfiltration. Data encoded in DNS queries; if DNS is allowed for any hostname, this works.
IPv6 fall-through. Policy covers IPv4; the tool uses IPv6.
Time-of-check vs. time-of-use. Hostname resolves to one IP at check time, another at connect time.
Egress proxy bypass. Direct outbound is supposed to be blocked at the network layer; misconfiguration allows it.

Defence: DNS allowlist (only specific hostnames resolvable); symmetric IPv6/IPv4 policy; DNS pinning at the egress proxy; verify the network layer blocks direct outbound.

Credential exfiltration¶

A compromised tool reads credentials from:

Environment variables (/proc/self/environ).
The cloud metadata service (169.254.169.254).
Files mounted by the runtime.
Memory of other processes (if isolation is incomplete).

Defence: scoped credentials minted at call time (chapter 06); metadata service blocked; per-call credential delivery via permission-restricted files; isolation layer that prevents cross-process memory reads.

Side channels¶

The hardware shared with other tenants or other tools is a side channel:

Cache timing. Data inferred from cache hit/miss patterns.
Branch prediction. Speculative execution leaks (Spectre, Meltdown class).
Resource contention. Workload inferred from observed contention.

Defences are imperfect:

Microcode updates for known speculative-execution vulnerabilities.
Co-location policy: high-stakes tools on dedicated hosts; multi-tenant tools on isolated capacity.
Cache partitioning where supported by the platform.

Side channels are a known limitation; defence is depth and acknowledgement, not elimination.

Supply chain¶

The sandbox depends on its base image, libraries, and runtime. A compromise in any of these propagates.

Defence:

Pinned versions of base images and dependencies.
SBOM (software bill of materials) for the sandbox itself.
Reproducible builds.
Security scans of images and dependencies.
Provenance verification (signed images, attestations).

A sandbox built from a compromised base image is a sandbox you do not have.

TOCTOU races¶

Time-of-check vs. time-of-use: the sandbox checks a path or condition; between check and use, the condition changes.

Examples:

The tool requests access to a file; the sandbox checks the path against the allowlist; between check and open, a symlink is changed to point elsewhere.
The tool requests network access to a hostname; DNS resolves to an allowed IP at check; the IP is reassigned to a different host before connect.

Defence: do the check atomically with the use where possible (openat with AT_SYMLINK_NOFOLLOW, capability-based access). For network, pin resolution at the egress proxy. Hardening tools (Capsicum, OpenBSD pledge) help.

Monitoring for escape patterns¶

Defence-in-depth includes monitoring. Even when a sandbox is escaped, the escape leaves traces:

Unexpected syscalls (calls denied by seccomp at high rate).
Network traffic to unexpected endpoints.
File access patterns outside the workspace.
Resource usage anomalies (a tool suddenly consuming far more than its profile).
Process spawning beyond expected.

The monitoring feeds the on-call apparatus's safety violation page (see 06_ai_runbooks_oncall chapter 03). An escape in progress should produce an alert before the damage compounds.

A worked example — the kernel CVE response¶

In 2024, a kernel CVE (CVE-2024-XXXX, illustrative) is published affecting container escape. The Hyderabad fintech's response:

Day 0. Security team receives the disclosure. Confirms the kernel is affected.
Day 1. Hot-patch applied to high-stakes hosts (payments, customer data tools).
Day 3. All container hosts patched after rolling update.
Day 7. A new microVM-based isolation for any tool not already on microVMs; the kernel surface for the most exposed tools is reduced.
Postmortem. The apparatus's response time was 1 day for critical hosts, 3 days for all hosts. The team adjusts the rapid-patch policy to target same-day for critical CVEs.

During the response window, escape monitoring (syscall denials, unusual process patterns) is heightened. No exploitation is observed.

Operational signals¶

Healthy. Vector-by-vector defences in place. Kernel CVEs patched within freshness budget. Escape monitoring fires on real anomalies. Quarterly review of the vector matrix.

First degrading metric. Kernel CVE patching lag climbing. The patching cadence is slipping; exposure window grows.

Misleading metric. Number of monitoring alerts. High volume from false positives produces alert fatigue; low volume can mean either secure or blind.

Expert graph. Vector × tool matrix with cell colour reflecting hardening; CVE patching latency; escape-pattern detection rate.

Boundary of applicability¶

Strong fit. Production sandboxes with non-trivial blast radius. The full vector defence is justified.

Pathology. Treating sandbox security as a one-time hardening. Vectors evolve; the matrix needs ongoing review. A sandbox that was hardened two years ago is now exposed to two years of new vectors.

Scale limit. Very large platforms have many sandbox configurations; the vector matrix grows. The pattern is a central security team owning the vector list and feature teams owning per-tool hardening.

Failure-prone assumption¶

The seductive wrong belief: a hardened sandbox cannot be escaped. It can. The history of container security is a history of escapes; the same will be true of microVMs and other isolation models. The correct belief: sandboxes are defence-in-depth with monitoring; an escape is caught and contained, not prevented absolutely.

Where this appears in production¶

A fintech patched a kernel CVE within 24 hours; the apparatus had pre-defined rapid-patch policy.
A telecom AI moved high-stakes tools to microVMs after a container escape postmortem.
A coding assistant had Docker socket mounted in a tool container; rebuilt without it.
A retail AI caught DNS exfiltration via syscall monitoring; tool retired.
A consumer chatbot had IPv6 fall-through; symmetric policy added.
A healthcare AI has microcode updates as part of regular patching; Spectre-class vulnerabilities tracked.
A legal AI has dedicated hosts for tools serving privileged client data; side-channel exposure reduced.
A government AI has SBOM for sandbox infrastructure; supply chain provenance verified.
A travel platform had a path traversal incident via symlink; canonicalisation added.
A logistics AI treats unexpected syscall denials as paging conditions; one such alert caught a probe.
A media AI runs reproducible builds; base image provenance is verifiable.
A staffing AI has TOCTOU defences via openat/AT_SYMLINK_NOFOLLOW.
A search-ops AI runs penetration tests against the sandbox quarterly; vectors are tested.
A document AI had a CVE in a parsing library propagate; dependency scanning added.
An ad-tech AI has co-location policy: high-stakes tools on dedicated hosts.
A real-estate AI monitors for cache-timing patterns; rate-limits to dampen side channels.
A B2B SaaS has CVE-driven patching cadence in their incident SLOs.
A small SaaS ignored a kernel CVE for two weeks; the next incident exploited it.
A medical AI has SBOM as a regulatory artefact.
A devops AI runs the sandbox infrastructure on a separate cluster from production; blast radius contained.

Recall / checkpoint¶

Name the eight common escape vectors.
What is the defence pattern for kernel CVEs?
What is the most common container misconfiguration that produces escape?
How does DNS exfiltration work, and how is it defended?
What is the TOCTOU pattern?
What does escape monitoring look like in practice?
Why is "a hardened sandbox cannot be escaped" a failure-prone assumption?

Interview Q&A¶

Q1. A critical kernel CVE is published affecting container escape. Walk through the response. Confirm the kernel is affected. Apply hot-patch to high-stakes hosts (payments, customer data, regulated workloads) within hours. Roll the patch out to all hosts within days. Heighten escape monitoring during the response window. Consider whether to migrate the most exposed tools to microVMs (separate kernel, lower exposure to host kernel CVEs). Postmortem: review patching SLO, adjust the policy if the response was slower than needed. Common wrong answer to avoid: "patch in the next maintenance window" — kernel CVEs warrant rapid-patch cadence.

Q2. The team uses containers with default seccomp. Walk through what hardening is still needed. Default seccomp blocks the most dangerous syscalls but allows many. Per-tool tightening: deny mount, umount, pivot_root, kexec, init_module, delete_module, ptrace, keyctl, bpf unless the tool genuinely needs them. Drop Linux capabilities to the minimum (--cap-drop all and selectively --cap-add). No Docker socket inside containers. Read-only root filesystem. AppArmor or SELinux profile on top. Each layer raises the escape cost. Common wrong answer to avoid: "default seccomp is enough" — default is conservative-broad; per-tool tightening pays off.

Q3. Walk through DNS exfiltration defence. DNS is often allowed when direct outbound is blocked. The tool encodes data in DNS queries: Q: AAA.BBB.attacker.com, where AAA and BBB carry exfiltrated bytes. Defence: DNS allowlist (only specific hostnames resolvable); the egress proxy is the DNS resolver and enforces the allowlist; verify that the tool's network namespace's DNS resolver points only at the proxy. Symmetric IPv6/IPv4 policy. Without DNS allowlist, the network policy has a hole. Common wrong answer to avoid: "block outbound, allow DNS for compatibility" — exactly the gap.

Q4. What is the TOCTOU pattern and how do you defend? Time-of-check vs. time-of-use: the sandbox checks a path or condition; between check and use, the condition changes. Example: tool requests /workspace/foo.txt; sandbox checks the path is in the allowlist; between check and open, the tool replaces foo.txt with a symlink pointing to /etc/passwd; the open follows the symlink. Defence: atomic check-and-use primitives (openat with AT_SYMLINK_NOFOLLOW), capability-based access, hardening libraries that integrate check with use. Common wrong answer to avoid: "the sandbox checks, so it's safe" — the check is necessary; atomicity is structural.

Q5. Side channels are imperfectly defendable. How do you treat them? Acknowledge the limitation. Defence: microcode updates for known speculative-execution vulnerabilities; co-location policy (high-stakes tools on dedicated hosts); cache partitioning where supported. Threat model: side channels typically require co-location and significant effort; the apparatus's other defences (scoped credentials, narrow policy, monitoring) limit what an exploit produces. Side channels are a residual risk to be managed, not eliminated. Common wrong answer to avoid: "side channels are a hardware problem" — they affect the apparatus's guarantees.

Q6. How does escape monitoring work, and what is its value? Monitor for patterns that indicate escape: unexpected syscall denials (seccomp blocks), unusual network endpoints, file access outside the workspace, resource anomalies, unexpected process spawning. Alerts fire on patterns; the on-call apparatus's safety-violation page is the channel. Value: catching escapes that succeed despite hardening. A defence-in-depth approach assumes some layer will fail; monitoring is how the failure is caught before damage compounds. Common wrong answer to avoid: "hardening is the defence" — hardening reduces probability; monitoring catches what hardening misses.

Design / debug exercise (10 minutes)¶

Modelled example. Walk through the worked example (the kernel CVE response). Identify the apparatus components engaged: rapid patching, escape monitoring, postmortem with policy adjustment.

Your turn. For your team's sandbox, walk through each of the eight vectors. For each, identify the defence in place and any gap. Highest-risk gaps are your next sprint.

Reproduce from memory. List the eight vectors and the typical defence for each. The signal of internalisation is that you can audit a new sandbox configuration against the matrix quickly.

Operational memory¶

This chapter explained the escape vectors a sandbox faces and the defences in depth. The important idea is that sandboxes are not unbreakable; the discipline is to know the vectors, harden against them, and monitor for the patterns that signal escape in progress.

You learned to recognise each vector, design the layered defence (rapid patching, syscall restriction, network allowlist, scoped credentials, monitoring), and treat the vector matrix as a living artefact. That solves the opening failure because the sandbox now has structural defence-in-depth across vectors.

Carry this diagnostic forward: when a sandbox is "secure" but unmonitored, you have found the next investment. Hardening reduces probability; monitoring catches what hardening misses.

Remember:

Eight vectors: kernel CVE, container escape, traversal, network bypass, credential exfil, side channel, supply chain, TOCTOU.
Defence-in-depth across vectors; monitoring catches what hardening misses.
Kernel CVEs warrant rapid patching, not maintenance-window patching.
Side channels are residual risk; manage, don't promise elimination.
The vector matrix is a living artefact; review quarterly.

Bridge. Escape vectors and defences are individual-tool concerns. Multi-tenant isolation is the cross-cutting concern — one tenant's tool calls must not affect another tenant. The next chapter is that discipline. → 10-multi-tenant-isolation.md