02. LLM Agent Orchestration Platform — Low-Level Design¶
~20 min read · Part 3 of 4 (Overview → HLD → LLD → Q&A)
The HLD named the boxes. This file opens the four that carry the design's weight — the planning loop, tool/function calling, the context/memory manager, and the guardrail-and-budget layer — and pins down the data, the algorithms, and the concurrency corners where an agent platform actually breaks.
Data models¶
The source of truth is an append-only event log per task. Current state is a projection over it; nothing else is authoritative.
CREATE TABLE task (
task_id UUID PRIMARY KEY,
tenant_id BIGINT NOT NULL,
goal TEXT NOT NULL,
status SMALLINT NOT NULL, -- 0=running 1=waiting_approval 2=done 3=failed 4=cancelled
budget_usd NUMERIC NOT NULL, -- hard ceiling, e.g. 1.00
spent_usd NUMERIC NOT NULL DEFAULT 0,
max_steps INT NOT NULL, -- e.g. 20
step_count INT NOT NULL DEFAULT 0,
replans_left SMALLINT NOT NULL DEFAULT 3,
created_at TIMESTAMP NOT NULL,
lease_owner TEXT NULL, -- worker id holding the task, null if free
lease_expiry TIMESTAMP NULL
);
CREATE TABLE step_event ( -- the log; one row per transition
task_id UUID NOT NULL,
step_n INT NOT NULL, -- monotonic per task
kind SMALLINT NOT NULL, -- 0=plan 1=tool_call 2=observation 3=final 4=error
payload JSONB NOT NULL, -- decision / args / result, model-visible
in_tokens INT NOT NULL DEFAULT 0,
out_tokens INT NOT NULL DEFAULT 0,
cost_usd NUMERIC NOT NULL DEFAULT 0,
created_at TIMESTAMP NOT NULL,
PRIMARY KEY (task_id, step_n) -- (task_id, step_n) is the idempotency key
);
Three deliberate choices. First, (task_id, step_n) is the primary key and the idempotency key: a worker that retries a step writes the same step_n, and the insert either lands once or conflicts, so a re-executed step can never double-append. Second, spent_usd and step_count live on the task row and are updated in the same transaction as the step insert, so the budget check and the spend record can never drift apart. Third, lease_owner/lease_expiry implement a lease, not a lock — if a worker dies, the lease simply expires and another worker takes over; there is no held lock to leak.
Tools are described by a registry the model sees as function schemas:
{
"name": "pricing_api",
"description": "Fetch current published pricing for a named vendor.",
"parameters": {
"type": "object",
"properties": { "vendor": {"type": "string"}, "region": {"type": "string", "enum": ["us","eu"]} },
"required": ["vendor"]
},
"policy": { "gated": false, "idempotent": true, "timeout_ms": 10000, "max_retries": 2, "rate_per_min": 60 }
}
The parameters block is exactly what goes to the model as the function schema; the policy block is server-only — the model never sees it and cannot override it. idempotent: true is what lets the executor cache identical calls and safely retry them; a non-idempotent tool (say charge_card) is marked gated and never auto-retried.
Long-term memory is a separate store, keyed for similarity rather than by task:
memory_record: { id, tenant_id, embedding[1536], text, source_task, created_at }
index: HNSW over embedding, filtered by tenant_id
Component internals¶
Component 1 — The planning loop (orchestrator run)¶
Responsibility: advance one leased task by one durable step, enforcing budget and turning both success and failure into the next observation. The loop is deliberately simple; all the intelligence about what to do next lives in the model, and all the intelligence about whether it is allowed lives in the guardrail.
def run_step(task): # one durable transition
if task.step_count >= task.max_steps or task.spent_usd >= task.budget_usd:
return finish(task, reason="budget") # hard ceiling, enforced by runtime
ctx = context_mgr.assemble(task) # compacted history + memory + schemas
decision, usage = llm_gateway.decide(ctx) # model picks: tool_call or final
record(task, kind=PLAN, payload=decision, usage=usage) # append + update spend, one txn
if decision.is_final:
return finish(task, answer=decision.text)
call = decision.tool_call
verdict = guardrail.check(task, call) # schema + policy + budget
if verdict.gated:
return park_for_approval(task, call) # status=waiting_approval, back to queue
if verdict.rejected:
observe(task, error=f"blocked: {verdict.reason}") # feed back, let model replan
return CONTINUE
result = tool_executor.run(call) # isolated, timed, retried per policy
if result.failed:
return handle_tool_failure(task, call, result) # see Component 2
observe(task, payload=result.value) # append observation
return CONTINUE
record, observe, and finish each append one step_event and update the task's spent_usd/step_count in a single transaction, so a crash between the model call and the log write leaves the task exactly where it was — the model call is retried, which costs one call but never corrupts state. The loop returns after each step so any worker can run the next one; a long task is thousands of these transitions, not one long-held thread.
Component 2 — Tool/function calling and failure recovery¶
Responsibility: dispatch the model's structured tool call to the right executor, run it in isolation with a timeout, retry within policy, and — the part that matters — convert an unrecoverable failure into an observation the model can plan against.
def run(call) -> ToolResult: # in the Tool Executor
tool = registry[call.name]
key = idempotency_key(call) # hash(name + args)
if tool.idempotent and (hit := result_cache.get(key)):
return hit # identical prior call, no re-execute
for attempt in range(tool.max_retries + 1):
r = sandbox.invoke(tool, call.args, timeout=tool.timeout_ms)
if r.ok:
if tool.idempotent: result_cache.set(key, r, ttl=300)
return r
if not transient(r.error): break # 4xx / bad-args: don't retry
sleep(backoff(attempt)) # transient: 5xx / timeout
return ToolResult(failed=True, error=r.error) # exhausted retries
def handle_tool_failure(task, call, result): # in the orchestrator
if task.replans_left > 0:
task.replans_left -= 1
observe(task, error=f"tool {call.name} failed: {result.error}. "
f"Consider an alternative approach.") # model replans
return CONTINUE
return finish(task, reason="tool_unrecoverable", partial=True)
The split is the point: the executor owns mechanical retries (same call, backoff, bounded), while the orchestrator owns strategic recovery (give up on this tool, tell the model, let it choose a different one). Retrying a timed-out call twice is the executor's job; deciding to abandon pricing_api for web_search is the model's, prompted by the observation the orchestrator injects.
Component 3 — Context / memory manager (compaction)¶
Responsibility: build the smallest prompt that still lets the model make the right next decision, so cost stays flat instead of climbing quadratically.
def assemble(task) -> Context:
fixed = system_prompt + tool_schemas # ~5k tokens, prompt-cached prefix
recent = last_k_events(task, k=3) # last 3 steps verbatim
older = summarize(events_before(task, k=3)) # bulky old tool dumps → 2-line extracts
recall = vector_store.topk(embed(task.goal), k=4, tenant=task.tenant_id)
scratch = task.scratchpad # running verbatim notes the agent keeps
ctx = fixed + task.goal + scratch + recall + older + recent
if count_tokens(ctx) > WINDOW * 0.9: # still too big → summarize harder
ctx = fixed + task.goal + scratch + hard_summary(task)
return ctx
The rule is keep what drives the next decision verbatim, compress what only provided evidence for decisions already made. An 8k-token raw search dump from step 1 becomes, by step 4, the two lines the agent extracted ("Top 3 competitors: A, B, C") — the model already used the dump, so it costs nothing to drop the rest, and it saves resending 8k tokens on every remaining step.
Core algorithm — the loop under the threaded scenario, with a tool failure¶
Walk the competitor-pricing task step by step, tracking context size and spend against the $1.00 / 200k-token budget, at $3/M input and $15/M output.
- Step 1 — plan + search. Context = 5k fixed + 1k goal = 6k in. Model emits
web_search("competitors to product X")(~0.5k out). Executor returns an 8k-token result; the context manager extracts "A, B, C" into the scratchpad. Spend so far:6k×$3/M + 0.5k×$15/M ≈ $0.026. - Step 2 — search pricing pages. Compacted context = 5k + goal + scratchpad + last step ≈ 9k in. Model calls
web_search("A B C pricing"), gets another 8k dump, extracted to a few lines. Spend cumulative ≈ $0.06. - Step 3 — parse with code. Context ≈ 11k in. Model calls
code_execto normalize the extracted snippets into a table skeleton; result 2k, kept. Cumulative ≈ $0.10. - Step 4 — pricing_api times out. Context ≈ 13k in; model calls
pricing_api(vendor="A"). The executor invokes with a 10 s timeout, gets nothing, retries twice with backoff, and after ~15 s of wall-clock returnsfailed. No LLM tokens were spent on the retries.replans_leftis 3, so the orchestrator appends the observation "pricing_api failed: timeout. Consider an alternative approach." and decrements to 2. Cumulative spend unchanged by the retries: still ≈ $0.14. - Step 5 — replan. Context ≈ 14k in (adds the failure observation). The model reads the failure and reaches for
web_search("A official pricing page")instead — the failure became an observation it planned against. Result extracted. Cumulative ≈ $0.19. - Step 6 — fill remaining vendors. Context ≈ 14k in; model calls
web_search/code_execto complete B and C pricing. Cumulative ≈ $0.24. - Step 7 — synthesize. Context ≈ 15k in; model writes the comparison table, ~3k output tokens.
15k×$3/M + 3k×$15/M ≈ $0.09. Cumulative ≈ $0.33.
The task finishes in 7 recorded steps (one extra beyond the planned 6 because of the replan), at ~$0.33 — well under the \(1.00 budget, and well under the ~\)0.97 the naive no-compaction version would have cost, because compaction held each call near 14k tokens instead of letting it climb to ~58k. This particular run also comes in under the ~$0.54 fleet estimate from the overview: that figure assumes a flat 14k tokens across a full 8 calls, whereas this task ran 7 steps with most contexts below 14k, so the per-task average and the worst-case fleet planning number legitimately differ — you budget capacity against \(0.54 and expect typical runs like this one to land lower. The failure cost ~15 s and one extra LLM call (~\)0.05), absorbed by the budget headroom that per-task budgets exist to provide. Had the tool stayed broken through three replans, the loop would have terminated with a partial table rather than burning to the ceiling.
Sequence diagram — a tool failure and recovery at step 4¶
Orchestrator Guardrail ToolExecutor pricing_api LLM Gateway Event log
│ step 4 │ │ │ │ │
├─ decide ───────┼───────────────┼────────────────┼──────────────▶│ tool_call │
│◀───────────────┼───────────────┼────────────────┼───────────────┤ pricing_api │
├─ check(call) ─▶│ │ │ │ │
│◀── allow ──────┤ │ │ │ │
├─ run(call) ────┼──────────────▶│ invoke ───────▶│ │ │
│ │ │ (10s ... timeout) │ │
│ │ │ retry×2 (backoff, ~15s) ──────▶│(still down) │
│◀── failed ─────┼───────────────┤ │ │ │
├─ handle_failure: replans_left>0 → observe("pricing_api failed…") ┼─────────────▶│ append
│ step 5 │ │ │ │
├─ decide ───────┼───────────────┼────────────────┼──────────────▶│ tool_call │
│◀───────────────┼───────────────┼────────────────┼───────────────┤ web_search │
├─ run(web_search) ─────────────▶│ invoke ───────────────────────▶ (ok) ────────▶│ append
│ ... loop continues to synthesis ...
The retries stay inside the executor; only the final failure crosses back to the orchestrator, which turns it into an observation and lets the model choose the alternative tool on the next step.
Concurrency and edge cases¶
- Duplicate step execution on worker crash: a worker that dies after running a tool but before appending the observation is resumed by another worker, which re-runs the step. Safe because tool calls are keyed by
idempotency_key(call)— an idempotent tool returns the cached result, and thestep_eventinsert is guarded by the(task_id, step_n)primary key, so the observation lands exactly once. - Budget race under concurrent steps: there is only ever one active lease per task, so steps for a task are serial by construction — no two workers advance the same task at once.
spent_usdis updated in the same transaction as the step insert, so the budget check reads a consistent number. - Non-idempotent tool + retry: a tool like
charge_cardmust never be auto-retried, because a timeout might mean the charge succeeded but the response was lost. Such tools are marked non-idempotent and gated; the executor never retries them, and recovery from their ambiguous failure goes through human approval, not an automatic replan. - Parallel tool calls in one step: when the model requests several independent tool calls at once (fetch three vendors' pages in parallel), the executor runs them concurrently and the orchestrator appends all results as one observation batch, but the step is still a single durable transition — partial success is recorded per call so a retry only re-runs the ones that failed.
- Lease expiry mid-tool: if a long tool call outlives the lease, the lease is renewed with a heartbeat while the call is in flight; a worker that cannot heartbeat (it is truly dead) loses the lease and its in-flight work is re-executed under the idempotency guarantees above.
- Stale memory recall: the vector store can surface an outdated fact into context. Because memories are advisory context and not authoritative state, a wrong recall degrades answer quality but cannot corrupt task state — the guardrail still validates every action the model takes on the strength of it.
- Compaction dropping something load-bearing: aggressive summarization can discard a detail the model later needs, causing it to re-fetch. This is a quality/cost tradeoff, not a correctness bug — the re-fetch costs a step; the fix is tuning what the scratchpad keeps verbatim, not disabling compaction, whose absence would nearly triple per-task cost.