06. Secrets and credentials in tools¶

~8 min read. Policy bounds where a tool reaches. Credentials bound what the tool can do where it reaches. The discipline is to mint scoped, short-lived tokens at call time — never to share the agent's credentials with the tool.

Continues from 05-filesystem-and-network-policy.md. This chapter develops the credential layer. Recurring concepts in bold: scoped credential, credential broker, short-lived token, least-privilege scope, per-call mint, credential exfiltration.

A tool that runs with the agent's full IAM role has the agent's full power. Compromising the tool compromises the agent. The credential layer is the structural defence against this amplification.

What scoped credentials look like¶

A scoped credential is a token, certificate, or IAM role that authorises a narrow set of operations on a narrow set of resources, for a short time.

Property	Why it matters
Narrow scope	If leaked, the damage is bounded to the scope
Short lifetime	If leaked, the window of exploitation is bounded
Per-call minted	No sharing across calls; revocation is per-call
Tenant-specific	Cross-tenant access is structurally impossible

A useful pattern: the tool's manifest declares the resource and operation set it needs. At call time, a credential broker mints a token with exactly those scopes for this call, valid for the call's wall-time cap plus a small margin. After the call, the token expires.

The credential broker¶

The broker is a small platform service that translates tool-manifest claims into provider-specific credentials.

Inputs: - The tool's identity (which tool is calling). - The call's identity (call ID, tenant ID, user ID). - The tool's scope claims from its manifest.

Outputs: - A scoped credential the tool can use to authenticate to the resources it needs.

The broker is the chokepoint. Tools never have raw credentials; they have broker-issued tokens. The broker enforces policy (which scopes are permitted for which tools), logs every issuance, and integrates with the platform's secrets management (Vault, AWS Secrets Manager, GCP Secret Manager).

What the agent's credentials must not have¶

The agent itself runs with credentials that allow it to dispatch tool calls and serve the model. These credentials must not include:

Direct access to customer data (only tools' scoped credentials should).
Direct access to payment, irreversible-action, or destructive operations.
The credentials to mint new credentials (the broker has those; the agent uses the broker).

The agent's credentials are narrow by design. A compromise of the agent must not yield the keys to the kingdom; the kingdom's keys live in the broker, behind authentication that the agent itself has.

A worked example — the data analysis tool¶

The Hyderabad fintech's data analysis tool needs:

Read access to the analytics database for the calling tenant only.
Read access to the tenant's data in S3 for the relevant datasets.
Write access to the tenant's per-call workspace.

The credential design:

At call time, the broker mints a short-lived token (30-minute lifetime) with:
Postgres credentials scoped to analytics-db with SELECT on the tenant's schema.
AWS credentials with s3:GetObject on s3://<bucket>/<tenant>/* and nothing else.
Filesystem credentials (effective UID/GID) that match the workspace owner.
The token is delivered to the tool via a credential file in the workspace, with file permissions 0600.
The tool uses the token for the call; on call completion, the token is revoked at the broker.

A compromised tool — adversarial input, hallucinated query, bug — can only do what the scoped token allows: read this tenant's analytics data and write to this tenant's workspace. It cannot reach other tenants' data, cannot modify the database, cannot reach the cloud provider's account-level resources.

The previous version's tool ran with the agent's IAM role, which had broad write access to multiple S3 buckets. The replacement is structurally narrower.

Credential rotation¶

Even scoped credentials need rotation:

Broker root credentials. The broker itself has credentials to mint tokens. Rotated quarterly with zero-downtime.
Per-tool credentials. If a tool's underlying credentials need rotation (e.g., the SSL certificate the broker uses to authenticate to the database), the rotation is coordinated.
User-facing credentials. If a user's credential changes (e.g., they leave the company), broker policy is updated; new tokens issued reflect the change.

Rotation is automated; manual rotation accumulates risk because it lags real events.

Common credential pitfalls¶

Hardcoded credentials in tool code or environment variables. The most common pattern; produces credentials that cannot be rotated without redeployment. Use broker-issued credentials instead.

Long-lived credentials. If a credential is leaked, the window of exploitation is the credential's lifetime. Short lifetimes (minutes to hours, not days) bound the window.

Broad scopes for convenience. "Give the tool read+write to everything in the tenant's bucket so we don't have to enumerate paths." The convenience is paid in incidents.

Credentials in tool output. A tool that writes its credentials to logs or returned data leaks them to whatever consumes the output. The output validation layer (chapter 08) should sanitise; tools should not include credentials in any output.

Sharing credentials across tenants. A tool that uses one credential to serve multiple tenants conflates them; one tenant's call can see another tenant's data through the shared credential.

Operational signals¶

Healthy. Every tool uses broker-issued, short-lived, narrowly-scoped credentials. The agent itself has no broad credentials. Broker logs every issuance and is audited regularly.

First degrading metric. Broker token lifetimes creeping up. The platform's discipline on short lifetimes is weakening.

Misleading metric. Number of tokens issued. High volume can mean healthy operation (short lifetimes, per-call minting) or a runaway loop minting unnecessarily.

Expert graph. Per-tool token scope and lifetime distribution; broker policy changes over time; credential exfiltration alert rate.

Boundary of applicability¶

Strong fit. Tools accessing scoped resources with explicit access models. The full scoped-credential pattern is justified.

Pathology. Tools that require very broad access (e.g., a "search anything in the company's data" tool). These warrant heavier scrutiny — narrower defence at other layers, more aggressive monitoring — rather than wider credentials.

Scale limit. Very large platforms have many tools, many tenants, many resources; the broker becomes a critical platform service. Pattern: federated brokers per cloud or per business unit; central audit.

Failure-prone assumption¶

The seductive wrong belief: environment variables are secret. They are not. Anyone with access to the running process can read them (/proc/<pid>/environ on Linux). Hardcoded credentials in environment variables are not protected; they are made convenient. The correct belief: credentials are minted at call time, delivered via secure channels (file with restrictive permissions, in-memory only), and revoked on completion.

Where this appears in production¶

AWS Lambda uses execution role credentials minted per invocation; long-lived credentials are not shipped with code.
A fintech has a credential broker for all tool credentials; tools never see long-lived secrets.
A telecom AI has per-tenant scoped credentials; cross-tenant access is structurally impossible.
A coding assistant had a tool with the agent's IAM role; tightened to per-tool scoped roles; incident exposure dropped.
A retail AI uses HashiCorp Vault as the broker; tokens are minted with TTLs measured in minutes.
A consumer chatbot had hardcoded API keys in tool code; rotated quarterly manually; missed two rotations; incident exposed.
A healthcare AI has the broker integrated with the identity provider; tokens are minted per user-on-behalf-of.
A government AI has scoped credentials audited for regulatory compliance; the broker is the audit subject.
A B2B SaaS has tool credentials short-lived (10 minutes); long calls renew at the broker.
A travel platform had a tool's credentials leak via a returned tool result; sanitisation added.
A legal AI uses workload identity (Google Cloud) — no long-lived credentials in any tool.
A logistics AI has the broker policy reviewed quarterly; each tool's scope is verified against its actual needs.
A search-ops AI uses certificate-based authentication; certificates are minted per call with short lifetime.
A media AI had a tool with broad S3 access; tightened to per-call signed URLs; access is per-object.
A staffing AI has a credential rotation game day quarterly; rotation lag is measured.
A document AI uses STS AssumeRole for AWS credentials; per-call short-lived.
An ad-tech AI has the broker as a security audit subject; access to the broker is itself MFA-gated.
A real-estate AI uses service mesh for in-cluster authentication; long-lived secrets are not in any tool.
A medical AI has credentials documented per tool for HIPAA audit.
A small SaaS has the agent's credentials shared with tools; the next incident scales to the agent's blast radius.

Recall / checkpoint¶

What are the four properties of a scoped credential?
What is the credential broker, and what does it produce?
What credentials must the agent itself not have?
List five common credential pitfalls.
Why are environment variables not secret?
What does credential rotation defend against?
How does this layer interact with the policy layer?

Interview Q&A¶

Q1. A team's tools use the agent's IAM role. Walk through the structural problem and the fix. The agent's role aggregates the permissions of all tools. Any tool compromise yields the role's full power; cross-tool privilege escalation is structural. The fix is the credential broker pattern: per-tool, per-call scoped credentials. The agent's role itself becomes narrow — it can call the broker and dispatch tools, nothing more. Each tool's blast radius is bounded by its scoped credential, not by the agent's. Common wrong answer to avoid: "rotate the agent's role frequently" — rotation does not bound blast radius.

Q2. Walk through the broker pattern. The broker is a platform service. Tools' manifests declare their resource and operation scopes. At call time, the broker mints a credential matching the manifest claims, valid for the call's wall-time plus a margin. The credential is delivered to the tool through a secure channel; on call completion, it is revoked. Logs every issuance for audit. Integrates with the platform's secrets management. The broker is the chokepoint; tools never see long-lived credentials. Common wrong answer to avoid: "the broker is overkill" — without it, credentials are managed per tool with no central enforcement.

Q3. Why are environment variables not a safe credential channel? Environment variables are readable by anyone with access to the process — /proc/<pid>/environ on Linux, similar elsewhere. A tool reading env vars to find its credentials is also exposing them to any escape vector that gets to the process. The correct pattern is broker-issued credentials delivered through file (with restrictive permissions) or in-memory (via API), never through env vars. Common wrong answer to avoid: "env vars are scoped to the process" — they are readable by privileged processes and may leak via misconfigurations.

Q4. The tool needs broad access (e.g., to search across all the tenant's resources). How do you scope? Acknowledge that some tools have broad needs and treat them with extra discipline. Narrow within the tenant (cross-tenant remains impossible). Narrow by operation (read but not write, list but not delete). Narrow by time (short TTL, renewed at the broker). Compensate with other layers: heavier isolation (microVM), aggressive monitoring (every call audited), approval gates for high-blast-radius actions. The broad scope is acknowledged; the other layers compensate. Common wrong answer to avoid: "give the tool the broad scope" — broad scope is the failure to avoid; compensation is the response.

Q5. Walk through credential rotation. Automated rotation at the broker. Short token lifetimes (minutes to hours) mean each token rotates by expiry, not by manual intervention. Broker root credentials rotate quarterly via a zero-downtime process. User-facing scope changes (someone leaves, role changes) are reflected at the next token issuance. Rotation is observable: a rotation that fails is alerted. Common wrong answer to avoid: "quarterly manual rotation" — manual rotation lags real events and accumulates risk.

Q6. How does the credential layer interact with the policy layer? The two compose. Policy bounds where the tool can reach (which paths, which endpoints, which syscalls). Credentials bound what the tool can do where it reaches (which operations are authorised). A tool with tight policy but broad credentials can act at the policy's boundary. A tool with broad policy but tight credentials can reach widely but do little. Defence-in-depth means both are tight. Common wrong answer to avoid: "credentials are enough" — credentials authorise at the resource; policy controls reach.

Design / debug exercise (10 minutes)¶

Modelled example. Walk through the worked example (the Hyderabad data tool's credential design). Verify the broker pattern, the scope, the lifetime, the revocation.

Your turn. Pick one tool. Describe its current credential model. Identify any long-lived credentials, broad scopes, or shared credentials. Design the scoped-credential replacement.

Reproduce from memory. Write the four properties of a scoped credential and the broker pattern. The signal of internalisation is that you can design credentials for a hypothetical new tool quickly.

Operational memory¶

This chapter explained the credential layer: broker-minted, per-call, narrowly-scoped, short-lived tokens that bound a tool's effective authority. The important idea is that credentials are minted per call by a broker, never shared with tools as long-lived secrets, and structurally bounded by scope and lifetime.

You learned to design the broker pattern, scope per-tool, rotate by short lifetimes, and avoid the common pitfalls (env vars, long lifetimes, broad scopes). That solves the opening failure because a compromised tool can only do what its scoped token authorises.

Carry this diagnostic forward: when a tool has long-lived or broad credentials, you have found the team's next credential hardening target.

Remember:

Scoped credentials: narrow, short-lived, per-call minted, tenant-specific.
Broker is the chokepoint; tools never hold long-lived secrets.
The agent itself has narrow credentials; the broker holds the rest.
Environment variables are not safe channels.
Rotation is automated; manual rotation lags real events.

Bridge. Credentials bound what a tool can do where it reaches. Some actions, even with scoped credentials, cannot be undone — money transfers, mass deletions, external sends. The next chapter is the approval layer. → 07-irreversible-actions-and-approvals.md