00. Design an LLM Agent Orchestration Platform¶

~20 min read · Level: advanced · Part 1 of 4 (Overview → HLD → LLD → Q&A)

Problem¶

An LLM agent orchestration platform runs autonomous agents that break a goal into steps, call tools to act on the world, observe the results, and loop until the goal is met. This is the machinery behind OpenAI's Assistants/Responses runtime, Anthropic's agent SDK loops, LangGraph, and the internal orchestrators that power products like Devin or a coding copilot's "agent mode." A user hands the platform a goal in natural language; the platform stands up an agent that plans, invokes a search tool, runs code, hits an API, reads the outputs back into its own context, and keeps going until it produces an answer — or hits a budget, a guardrail, or a wall it cannot get past.

What makes this a genuine system-design problem, and not just "call a model in a loop," is the shape of the work. Each step is a slow, expensive, non-deterministic LLM call in the low-single-digit seconds. The steps are serial — step N+1 depends on what step N observed — so a six-step task is six calls stacked end to end, not six calls in parallel. Every step drags the entire growing conversation back into the model, so cost climbs faster than step count. And the tools fail: a search times out, an API returns a 500, code throws. The agent has to notice, recover, and keep making progress without either giving up or spinning forever. Get those three facts straight — serial latency, super-linear cost, and mandatory failure recovery — and the architecture follows.

To keep the reasoning concrete, thread one scenario through the whole design: an agent asked to "find our product's top three competitors, pull their current pricing, and build a comparison table." It is a six-step task — two web searches, a code-execution step to parse results, a pricing-API call, a second search, and a final synthesis — running under a hard budget of $1.00 and a 200k-token context window**, on a model priced at **$3 per million input tokens and $15 per million output tokens. Midway through, at step 4, the pricing API times out, and the agent must recover: retry, then fall back to a different tool, without blowing the budget or losing the work it has already done. That one task, that one budget, and that one failure will test every decision below.

Functional requirements¶

Run a multi-step agent loop: accept a goal, plan, call tools, observe results, and iterate until done or stopped.
Tool / function calling: expose a registry of typed tools (search, code execution, HTTP APIs), let the model request a call with structured arguments, execute it in isolation, and feed the result back.
Context and memory management: assemble the right context into each model call, compact it as it grows, and persist long-term memory the agent can retrieve across steps and sessions.
Guardrails and budgets: enforce per-task limits on steps, tokens, cost, and wall-clock time; validate tool calls before executing; support human-in-the-loop approval for dangerous actions.
Durable, resumable execution: checkpoint task state so a crash mid-task resumes rather than restarts, and long tasks can run asynchronously while the client streams progress.

De-scoped for this round, and worth naming so the interviewer hears it as a choice rather than a gap: training or fine-tuning the models themselves, a visual agent-builder UI, multi-agent negotiation protocols (we support sub-agents as a tool, not a full agent society), and marketplace billing. These matter, but they sit beside the core loop and do not change its architecture.

Non-functional requirements¶

The dominant constraint is end-to-end tail latency of a serial, stateful loop — and the cost that rides on it. Because steps are sequential LLM calls, latency and cost both scale with step count, and every architectural decision is really a decision about how to keep a loop bounded.

Latency: a single LLM call runs ~2–4 s; a six-step task is therefore ~20–40 s end to end before tool time. The platform must stream partial progress so the task feels alive, and must never make the client wait synchronously for a minute-long task.
Cost: because the full context is resent on every step, cost grows super-linearly with steps. Cost is a first-class SLO, not an afterthought — a runaway loop is a runaway bill.
Reliability of progress: the loop must always terminate — on success, on budget, or on a guardrail — and must recover from tool failures rather than crashing the task.
Durability: a task's step history is the source of truth. Losing it mid-run wastes real money already spent on completed steps, so state must survive an orchestrator crash.
Isolation: tools run untrusted code and hit external networks. A tool must not be able to exhaust the host, escape its sandbox, or take down the orchestrator.

Scale estimation¶

Assume a mid-to-large platform: 1,000,000 agent tasks per day, each averaging 6 steps (so ~7 LLM calls counting the final synthesis).

Task throughput is 1,000,000 / 86,400 s ≈ 11.6 tasks/second on average. Apply a 10× peak factor and design for ~116 tasks/second at peak. LLM calls run at 11.6 × 7 ≈ 81 calls/second average, ~810 calls/second at peak — that is the load the model gateway must shape and rate-limit.

The number that actually sizes the fleet is concurrent in-flight tasks, because tasks are long-lived. At ~40 s average duration, concurrency is 11.6 tasks/s × 40 s ≈ 464 tasks in flight on average, and ~4,640 at peak. These are not threads blocking on CPU; they are mostly parked waiting on an LLM or a tool, so the orchestrator must be built around suspended, checkpointed tasks rather than a thread per task.

Now the cost, using the threaded task. Each call carries a fixed 5k tokens of system prompt plus tool schemas, and each completed step appends roughly 1.5k tokens of model output plus 6k tokens of tool result — ~7.5k tokens onto the history. If we naively resend the whole history every call, input tokens across 8 calls sum to 6k + 13.5k + 21k + 28.5k + 36k + 43.5k + 51k + 58.5k ≈ 258k tokens. At $3/M that is **$0.77 of input, plus ~13.5k output tokens at $15/M ≈ **$0.20, for ~$0.97 per task — right against the $1.00 ceiling, and a task that runs 12 steps instead of 6 blows straight through it, because the cost curve is quadratic in steps, not linear.

Reconcile that against the fleet: 1M tasks/day × ~$0.97 ≈ **$970k/day in model spend if we do nothing clever. This is why context management is not a nicety — it is the line item. Compacting the working context to a flat ~14k tokens per call drops input to 8 × 14k = 112k tokens ≈ $0.34, total ~$0.54/task**, roughly **$540k/day — a $430k/day** difference from one design decision. For storage, each task's event log runs ~200 KB of text; at 1M/day that is ~200 GB/day, and a 30-day retention window is ~6 TB — modest, and not the interesting scaling problem. The interesting problem is the loop's cost and latency, not the bytes at rest.

API sketch¶

POST /v1/tasks
  body: { "goal": "find top 3 competitors and their pricing, build a table",
          "tools": ["web_search","code_exec","pricing_api"],
          "budget": { "usd": 1.00, "max_steps": 20, "max_tokens": 400000 } }
  202:  { "task_id": "t_9f2", "status": "running", "stream_url": "/v1/tasks/t_9f2/events" }

GET /v1/tasks/{task_id}/events            # server-sent stream of steps
  event: step   data: { "n": 4, "action": "tool_call", "tool": "pricing_api", ... }
  event: token  data: { "text": "Based on the three vendors..." }   # streamed synthesis
  event: done   data: { "status": "succeeded", "cost_usd": 0.61, "steps": 8 }

GET  /v1/tasks/{task_id}                   # current state + full step log
POST /v1/tasks/{task_id}/approve           # human-in-the-loop: approve a gated tool call
POST /v1/tasks/{task_id}/cancel            # hard stop; releases the budget

Solutioning¶

Start from the loop and the system shapes itself. An agent step is read the accumulated state, ask the model what to do next, do it, record what happened — which is an event-sourced state machine, not a request/response handler. So the core is an orchestrator that owns a durable step log per task, advancing one step at a time and checkpointing after each. This is the first reframe worth carrying into the room: an agent task is not a long request; it is a durable workflow. Treating it as a request means a dropped connection or a crashed process throws away real money already spent on completed steps; treating it as a checkpointed workflow means any worker can resume a parked task from its log. The 4,640 concurrent tasks at peak are mostly suspended state in a store, not live threads.

The first defining tradeoff is context window versus cost, and it is sharper than it looks. The naive read is that the 200k-token window is the constraint — the agent runs out of room. It doesn't: our six-step task tops out around 60k tokens, nowhere near the wall. The real constraint is that resending the growing history every step drives cost up quadratically, so you run out of budget long before you run out of window. The resolution is a context manager that compacts aggressively — it keeps the goal and a running scratchpad verbatim, but replaces bulky raw tool outputs (an 8k-token search dump) with the two lines the agent actually extracted, and it caches the fixed 5k-token system-and-schema prefix so repeated reads of it bill at a fraction. That is the $0.97 → $0.54 move, and at fleet scale it is $430k/day.

The second tradeoff is autonomy versus control. A more autonomous agent — free to call any tool, loop as long as it likes — solves harder tasks but is also the one that spends $50 in an infinite loop or executes a destructive API call it shouldn't. The resolution is a guardrail layer that sits between the model's decision and the world: every proposed tool call is validated against its schema and a policy, every task carries a hard step/token/cost budget that the orchestrator enforces (not the model — the model cannot be trusted to enforce its own limits), and a small set of dangerous tools require human approval before they fire. Autonomy inside a fence, with the fence enforced by the runtime.

The third tradeoff is multi-step latency and cost versus answer quality, and it collides head-on with failure recovery in our scenario. When the pricing API times out at step 4, the cheap move is to give up or to retry forever; the correct move is bounded recovery — retry the tool a fixed number of times with backoff, and if it stays down, feed the failure back to the model as an observation so it can replan and reach for a different tool (a web search of the pricing page). That recovery costs ~2 extra LLM calls and ~15 s, which is exactly why budgets are per-task and generous rather than per-step and tight: the system should be willing to spend a little more to route around a failure, but never willing to spend without limit. The last reframe: a failed tool call is not an error to abort on; it is an observation to plan against — the same feedback loop that drives normal progress drives recovery.

The result is a platform whose center is a durable, checkpointed orchestration loop; whose cost is governed by a compacting context manager and prompt caching; whose autonomy is fenced by a guardrail-and-budget layer the runtime enforces; and whose tool calls run in an isolated executor that turns failures into observations the agent can plan around. The next files take each of these down to components (HLD) and then to schemas, the loop algorithm, and the concurrency corners (LLD).