09. Feedback cadence¶

Privacy ensures the data is handled responsibly. Cadence ensures it is used. A platform that captures feedback and never reads it is operating-blind with extra storage. The discipline is a rhythm of looking and acting on the appropriate horizon.

A platform engineer at a Chennai SaaS company captures rich feedback for six months without a regular review cadence. Insights accumulate; nothing is acted on; the team feels "we have all this data and don't use it." She establishes three cadences: weekly review of negative feedback (30 minutes; one person), monthly calibration cycle (judge calibration against the calibration set), quarterly cross-team retrospective (the broader team reviews patterns and sets priorities for the next quarter). Six weeks in, the eval set has grown by 80 cases, two prompts have been iterated, and the judge has been recalibrated. The cadence converted captured signal into actioned signal.

This chapter is that rhythm.

The three cadences¶

Cadence	Activity	Output
Weekly	Negative-feedback review; convert to eval cases	New eval cases; immediate prompt or system flags
Monthly	Judge calibration cycle	Judge refinements; eval baselines
Quarterly	Cross-team retrospective	Strategic priorities; backlog for next quarter

Each operates on a different horizon; together they cover the spectrum of action.

Weekly negative-feedback review¶

A 30-minute weekly session. One reviewer (a domain expert or PM, with engineering support).

Agenda:

Pull the past week's negative feedback (thumbs-down + critical comments + flagged implicit signals).
Sample 30-60 events.
Read each: input, response, user reaction.
Triage per chapter 05: skip, add to eval set, flag for prompt iteration, escalate as incident.
Output: a small list of additions to the eval set, a list of prompt or system flags, occasionally an incident escalation.

The session is short and recurring. The discipline is to do it consistently; one missed week is fine; six missed weeks is the queue backlog that produces "we have all this data and don't use it."

Monthly calibration cycle¶

The judge calibration from chapter 06. Monthly cadence; 2-3 hours of work distributed across reviewers.

Activities:

Run the judge against the calibration set.
Compute agreement; review disagreement cases.
If misalignment, refine rubric or judge prompt.
Re-validate.
Ship refinements; bump judge version; re-baseline.

The cycle is more substantial than the weekly. It produces fewer-but-deeper changes; the judge tracks user perception over time.

Quarterly cross-team retrospective¶

Once per quarter; 60-90 minutes; broader attendance (the agent team, product, customer-success, sometimes leadership).

Agenda:

Trends in feedback over the quarter: rates, patterns, segment differences.
Major eval-set additions and their rationale.
Prompt iterations and their outcomes.
Model migrations and their impact on feedback.
Cross-team priorities for next quarter: which segments to invest in, which feature areas to focus on.
Open issues: feedback patterns that have not yet been actioned.

The retrospective converts the operational rhythm of weekly and monthly into strategic alignment. Without it, the operational work proceeds without the team's bigger picture being calibrated.

Why each cadence matters¶

Weekly catches the immediate signal — new failure modes, specific cases the system did poorly on. It feeds the eval set continuously.

Monthly keeps the measurement instrument (the judge) aligned with reality. Without it, the eval scores drift from user perception.

Quarterly ensures the operational work serves the broader product strategy. Without it, the team optimises locally without cross-team awareness.

Skipping any cadence has predictable consequences: - Skip weekly: the eval set ages. - Skip monthly: the judge drifts. - Skip quarterly: the operational rhythm runs without strategic direction.

What to track between cadences¶

A small set of metrics on the dashboard, monitored daily by on-call or weekly by the lead:

Feedback volume per day per type
Negative-feedback rate per feature
Implicit-signal anomalies (chapter 03's signals — abandonment, repeat-ask)
Calibration agreement (from the monthly cycle; reported daily)
Pipeline throughput (cases added to eval per week)

The metrics are awareness; the action happens in the cadenced sessions.

When to break cadence¶

Sometimes a signal demands action between cadences:

A sudden spike in negative feedback on a specific feature (incident; chapter 11).
A judge-calibration result that shows significant drift (urgent refinement).
A regulatory event that changes what counts as acceptable feedback handling.

The cadences are the default; events that warrant immediate action override.

Common mistakes¶

No cadence. Captured feedback never gets used.

Inconsistent cadence. Skipped weeks compound into months of backlog.

Cadence without action items. Sessions happen but the team does not commit to changes.

Quarterly retrospective without trends. No data on what changed; the retrospective is opinion-driven.

Weekly review by an unprepared reviewer. The reviewer does not know the system well enough to triage; cases are added to the eval set indiscriminately.

Interview Q&A¶

Q1. Why three cadences instead of one? Different horizons. Weekly catches the immediate signal — new failure modes. Monthly maintains the judge alignment over time. Quarterly aligns the operational work with strategic direction. One cadence at one horizon misses the other two; weekly-only misses judge drift; monthly-only misses immediate failures; quarterly-only misses both. The three layers together cover the spectrum of necessary action. Wrong-answer notes: "one weekly meeting handles everything" produces calibration drift and strategic disconnection.

Q2. The team has captured feedback for six months without a cadence. What do you set up first? The weekly review. It is the lowest-effort highest-leverage cadence; it converts the immediate signal into eval cases. 30 minutes, one reviewer, one session. After two weeks, the team feels the pipeline working. Then add the monthly calibration cycle. Then the quarterly retrospective. The order is "immediate value first; longer horizons follow." Wrong-answer notes: starting with the quarterly retrospective produces opinion-driven sessions without operational rhythm.

Q3. The weekly review keeps getting skipped because "people are busy." What do you do? Make it ritual. A standing 30-minute calendar invite; one accountable owner who runs it; if the owner is out, a backup runs it. The output (cases added, flags raised) is visible in a shared document; missed weeks are visible. Skipping is a deliberate choice with consequences, not a default. After enough consistency, the ritual feels normal; the cost of skipping (the queue grows) is acknowledged. Wrong-answer notes: "ask people to prioritise" without structural support fails.

Q4. Between cadences, what should the team monitor? A small set of metrics on the dashboard: feedback volume, negative-feedback rate per feature, implicit-signal anomalies, calibration agreement, pipeline throughput. Daily or weekly checks (depending on platform scale). The metrics are awareness — they tell the team when a signal demands action between cadences. The action happens in the cadenced sessions or in escalated response (chapter 11). Wrong-answer notes: "no monitoring between cadences" misses the urgent signals.

What to do differently after reading this¶

Establish the three cadences as standing rituals.
One accountable owner per cadence; backup when the owner is unavailable.
Track between-cadence metrics so urgent signals are visible.
Allow cadence breaks for genuine incidents.
Make the output of each cadence visible to the broader team.

Bridge. Cadence is the rhythm. The next discipline is closing the loop — feeding the signals into the prompts, the model selection, the eval set decisions. The actual work the cadences produce. → 10-closing-the-loop.md