09. Coordinated multi-change releases¶

Change windows govern when individual changes ship. Coordinated multi-change releases happen when several changes ship together — a new feature with a prompt, a tool, and a retrieval index. The coordination is its own discipline.

A platform engineer at a Chennai SaaS company ships a new feature: a refund-explanation flow. The release includes a new prompt, a new tool, an updated retrieval index for refund policies, and a new eval set covering the flow. Each piece has its own discipline; together, the release requires coordination. The team plans: the retrieval index ships first (data change with schema check, sized for staged rollout); the new tool ships next (agent code change with integration tests); the new prompt ships behind a feature flag; the eval set is updated with cases for the flow. The canary on the new prompt is gated by the eval (already updated) and the feature flag (controls user exposure). Rollback for the multi-change release is documented per-change.

This chapter is the coordination discipline.

When multi-change is needed¶

Three common patterns.

A new feature with multiple AI artefacts. A new conversational flow needing prompts, tools, eval cases.

A correlated dependency change. A model migration that requires prompt revisions to behave correctly with the new model.

A platform refactor. Changes across multiple agents that must ship together for consistency.

Each requires multiple changes; the question is whether they ship as one coordinated release or as a sequence of independent releases.

Coordinated vs sequenced¶

The trade-off:

Coordinated. All changes ship together; the user sees the consistent state at once. Risk: rollback is complex; multiple change types in flight.

Sequenced. Changes ship one at a time, each with its own canary. Risk: intermediate states may be incoherent; users see partial functionality.

For most platforms, sequenced is the default; coordinated when the changes are tightly coupled (the new prompt requires the new tool to function).

The coordination plan¶

For a multi-change release, the plan:

Per-change discipline. Each change still follows its type's discipline (chapter 02). The eval gate, the canary, the rollback path per change.
Ordering. Which change ships first? Usually the foundation changes (data, tool) before the consumer (prompt that uses them).
Feature flag. A user-facing flag gates the new feature; the underlying changes can ship without exposing the feature.
Combined canary. The feature flag enables the feature for the canary percentage; the underlying changes are already at 100% (foundation) or canary'd in coordination.
Combined rollback. What does rollback look like? Often the feature flag drops to 0 (disabling the user-facing feature); the underlying changes remain in place.

Feature flags as the coordination tool¶

The feature flag is the per-user gate that decouples the underlying changes from the user-facing exposure:

features:
  refund_explanation:
    flag: refund_explanation_v1
    rollout:
      canary_percentage: 5
    dependencies:
      - prompt: explain_refund@1.0.0
      - tool: lookup_refund_eligibility@1.0.0
      - retrieval_index: refund_policies@2026-05-20

The flag enables the feature for the canary percentage; the dependencies are listed so the feature is enabled only when all are deployed. Rollback drops the flag to 0; users no longer see the feature; the underlying changes remain available for re-enable.

The order of operations¶

A reasonable sequence:

Foundation changes ship first. Data, tools, eval cases. These can ship without enabling the feature (data is in the index; tool is callable; eval cases are in the set but the feature is not yet exposed).
Prompt changes ship next. The prompt registry is updated; the new prompt is at weight 0 (no traffic yet).
Feature flag enabled at canary percentage. 5% of users see the feature; the prompt is at 5% on the route; the feature works end-to-end for those users.
Monitoring at each canary step. Feedback on the feature; per-component metrics (tool error rate, retrieval freshness, prompt response quality).
Promotion in canary steps. Each step is a feature flag percentage increase; the underlying changes may or may not increase their canary weights correspondingly.
Post-promotion monitoring. Standard 24-48h.

The coordination is in the ordering; each change's discipline operates per type.

Rollback for coordinated releases¶

The rollback is per-change, coordinated:

Feature flag. The fastest rollback: drop the flag to 0; users no longer see the feature. Underlying changes remain available.
Per-change rollback. If an underlying change is the issue (e.g., the new tool has a bug), rollback that specific change; the feature flag may remain at the canary percentage or drop to 0.
Cascading rollback. Rolling back the foundation change may require rolling back the prompt that depends on it.

The rollback decision is by change type: which change is the issue; what does rollback of that change look like; do other changes need to follow.

When coordination fails¶

If a coordinated release encounters issues:

Identify which underlying change is the cause.
Rollback that change first (or drop the feature flag to 0 if the cause is unclear).
Stabilise; investigate per the postmortem discipline.
Plan the re-release: fix the underlying issue; re-canary; promote with care.

The coordination's risk is that issues are harder to isolate; the discipline is to keep the changes individually identifiable so rollback can be surgical.

What multi-change does not solve¶

Inherent coupling. Some changes truly cannot ship independently; sequencing is forced.
Cross-team coordination. If different teams own the changes, the coordination is human-organisational, not just technical.
Risk amplification. A multi-change release is riskier than the sum of its parts; canary and post-promotion monitoring are heavier.

Common mistakes¶

Coordinated when sequencing would work. Risk amplified for no good reason.

Sequencing when coordinated is needed. Intermediate states incoherent; users see partial features.

No feature flag for user-facing changes. Rollback is complex; user exposure cannot be controlled independently.

No per-change rollback path. A complete rollback requires reverting everything; surgical fixes impossible.

Cross-team coordination via informal channels. Slack messages and verbal agreements; the multi-change release is undocumented.

Interview Q&A¶

Q1. When should changes ship coordinated vs sequenced? Sequenced by default. Coordinated when the changes are tightly coupled — the new prompt requires the new tool to function; the new feature requires data, tool, and prompt all to be in place. The trade is consistency (coordinated provides cleaner user experience for new features) vs risk (coordinated amplifies risk because multiple changes are in flight). For most platforms, sequencing with feature flags provides much of the coordination benefit at lower risk. Wrong-answer notes: "always coordinate" amplifies risk; "always sequence" produces incoherent intermediate states.

Q2. Walk through how a feature flag coordinates a multi-change release. The flag is a per-user gate on the user-facing feature. The underlying changes (prompt, tool, retrieval index) ship without enabling the feature. The flag is enabled at canary percentage (5%); the underlying changes coordinate to be available for those users. The feature is observable end-to-end for the canary; metrics on the feature inform promotion. Rollback drops the flag to 0; users lose the feature; the underlying changes remain available for re-enable. The flag decouples user exposure from artefact deployment. Wrong-answer notes: "ship everything to 100% then flag-on" misses the coordination value during canary.

Q3. The coordinated release encounters issues. How do you decide which change to rollback? Investigate which change is the cause. Read the production cases that exhibit the issue; isolate to one change if possible. If the cause is clear (the new tool's error rate is up), rollback that change; the feature flag may stay at the canary percentage if the rest of the feature works without it. If the cause is unclear, drop the feature flag to 0 immediately (containment); investigate root cause from a stable state. The discipline is "contain first, isolate the cause second." Wrong-answer notes: "rollback everything" is the broad action but loses surgical capability when one change is fine.

Q4. The team's tool team is shipping a new tool alongside the agent team's new prompt. How is the cross-team coordination structured? Explicit plan: each team owns their change; the ordering and timing are jointly agreed; the canary is coordinated; the rollback is jointly designed. The plan is documented (not Slack); it lists each change's owner, version, dependency, and rollback path. Communication during the release coordinates the steps. Postmortems are joint. The discipline turns cross-team coordination from informal into structured; the multi-change release is a documented operation, not a verbal handshake. Wrong-answer notes: "we'll figure it out as we go" produces the informal-coordination failure.

What to do differently after reading this¶

Default to sequenced; coordinate when changes are tightly coupled.
Feature flags as the coordination tool for user-facing changes.
Foundation changes first (data, tool); consumer changes (prompt) second.
Per-change rollback paths; rollback surgically when possible.
Cross-team coordination is documented and explicit.

Bridge. Coordinated releases are the planned multi-change discipline. Emergency changes are the unplanned override discipline. → 10-emergency-changes.md