Skip to content

02. Engineering Principles — Narrative Explainer

Companion to 03_study_material.md. The study material gives the frameworks. This explainer gives the judgment behind those frameworks.

Table of contents

ELI5 — the captain and the compass

Imagine a big ship crossing a difficult sea. Every sailor knows one useful skill. One sailor ties knots quickly. One sailor reads the wind well. One sailor patches torn sails. Those are individual engineering skills. Previous modules taught those skills one by one. Now you are not just a sailor anymore. You are the captain. The captain does not only pull ropes. The captain decides when to turn the ship. The captain balances speed against safety. The captain decides whether this storm is worth fighting. The captain keeps the crew calm and aligned. Engineering principles are the captain's decision framework. They tell you how to choose, not just how to code. Think of the technical direction as the course. Think of the decision framework as the compass. Think of the team and stakeholders as the crew. Think of risk assessment as the weather check. Think of documentation and ADRs as the ship's log. A weak captain has brilliant sailors and still crashes. A strong captain makes average sailors move like one mind. That is why principles matter more at Lead level. | Placeholder | Meaning | Why it matters | |---|---|---| | the course | Technical direction | Prevents random feature-driven wandering | | the compass | Decision framework | Keeps choices consistent under pressure | | the crew | Team and stakeholders | Aligns product, engineering, legal, and ops | | the weather check | Risk assessment | Stops reckless launches in risky conditions | | the ship's log | Documentation and ADRs | Preserves memory after people change |

Suppose the sea is calm and the deadline is near. You may choose a fast API and ship quickly. Suppose the sea becomes rough with privacy constraints. You may choose an open model and slower rollout. Suppose the crew starts arguing about direction. You open the ship's log and revisit the decision. Nothing magical happened there. You simply used principles before panic. That is the whole module in one picture. Good Leads are calm captains with a tested compass. Bad Leads keep changing course whenever a loud sailor speaks. Remember the ship whenever the later chapters feel abstract.

Chapter 1: The opening failure

1.1 Sprint 1 feels heroic

You join as the new Lead AI Engineer. The team is hungry, talented, and slightly overconfident. Sprint 1 feels fantastic. Features are shipping every few days. Demos look smooth in product review. Nobody wants to slow down for docs. Nobody wants to write tests for prompts. Everyone says they will clean it later. This is the first lie fast teams tell themselves. By Sprint 2, a second model is introduced. By Sprint 3, retrieval is patched into production. By Sprint 4, three people have changed prompts silently. No ADR exists for those changes. No one remembers why the first model was chosen. No one knows the fallback path when quality drops. Velocity still looks high on the board. Under the board, confusion is compounding. 

Sprint 1: Ship fast -> applause
Sprint 2: Add hacks -> applause
Sprint 3: Hidden coupling -> mild confusion
Sprint 4: Ownership blur -> repeated debates
Sprint 5: Incidents weekly -> everyone calls it complexity

1.2 Sprint 5 feels cursed

Sprint 5 looks nothing like Sprint 1. The system now feels tangled. Onboarding takes weeks, not days. Production breaks every week. Incidents take longer because logs are incomplete. Quality falls, but nobody trusts the evals either. Every design review reopens old arguments. The team is not stupid. The team is suffering from missing principles. Without principles, every urgent ticket becomes architecture. Without principles, every engineer invents private rules. Without principles, local optimization beats system quality. That is why the work feels cursed. It is not a talent problem. It is a decision system problem. | Sprint symptom | Hidden cause | |---|---| | Weekly prompt regressions | No review standard for prompt changes | | Same debate every planning meeting | No ADRs or decision criteria | | Long onboarding | Tribal knowledge, weak docs | | Hotfixes every Friday | No quality gates, no rollback discipline | | Unclear ownership | Team process optimized for demos, not systems |

1.3 Why this separates seniors from Leads

A strong senior engineer solves hard technical problems. A Lead engineer prevents the same problem from recurring. A senior optimizes a component. A Lead optimizes the decision system around components. A senior can ship despite ambiguity. A Lead must reduce ambiguity for the whole crew. That is the stakes of this module. Principles distinguish craftsmanship from technical leadership. They also distinguish heroics from repeatability. When interviewers ask leadership questions, this is what they test. They want to know whether you can choose the course. They want to know whether your compass works during storms. If you remember only one opening lesson, remember this. Chaos scales faster than code. Principles are the brake against that chaos.

Chapter 2: Technical decision-making

2.1 Start with the decision, not the tool

Most immature teams start with tool names. Mature teams start with decision questions. Do we need speed, control, or differentiation? What risk is acceptable for this release? How reversible is this choice? What will we need to operate later? These questions sound boring. They save months. A fancy tool without a decision frame becomes random complexity. A modest tool inside a strong frame becomes leverage. Begin with the problem shape. Then choose the technology shape. That ordering is a principle. 

Ask first:
1. What job must the system do?
2. What constraints are non-negotiable?
3. What is the cheapest reversible path?
4. What evidence would make us change later?
Then choose the stack.

2.2 Build vs buy vs fine-tune

This is the classic captain decision. Do you charter a ship, repair one, or build your own? In AI, the rough choices are buy, adapt, or build. Buy usually means API first. Adapt usually means prompt engineering plus RAG. Build usually means fine-tuning or self-hosted models. The beginner mistake is choosing the most prestigious option. The Lead chooses the option with the best learning-to-risk ratio. | Option | Speed | Control | Ops burden | Best default use | |---|---|---|---|---| | Buy API | Fastest | Lowest | Lowest | Prototypes, uncertain use cases | | Prompt + RAG | Fast | Medium | Medium | Domain grounding without retraining | | Fine-tune | Slow | Higher | Higher | Stable, repeated patterns with clear data | | Self-host open model | Slowest initially | Highest | Highest | Privacy, cost, or platform control |

Start with buy when uncertainty is high. APIs help you learn the real task quickly. Move to RAG when domain knowledge is the bottleneck. Move to fine-tuning when behavior is stable and data is strong. Move to self-hosting when economics, privacy, or platform leverage demand it. This order avoids expensive self-deception. 

Need an answer this quarter?
├─ Yes
│  ├─ Privacy and compliance allow external API?
│  │  ├─ Yes -> Buy first
│  │  └─ No  -> Evaluate open model or managed private deployment
│  └─ Quality issue is missing domain context?
│     ├─ Yes -> Add retrieval before fine-tuning
│     └─ No  -> Improve prompt, workflow, or tools
└─ No
   ├─ Is this capability strategic and repeated?
   │  ├─ Yes -> Consider fine-tuning or platform build-out
   │  └─ No  -> Buy and keep learning

Fine-tuning is not the first medicine for every pain. Sometimes the disease is weak prompts. Sometimes the disease is poor retrieval. Sometimes the disease is a broken workflow around the model. Leads diagnose before prescribing. That sounds obvious. It is repeatedly ignored in real teams. | Question | If answer is yes | If answer is no | |---|---|---| | Is the problem still poorly understood? | Buy first | You may invest more deeply | | Is domain knowledge the main gap? | RAG first | Consider workflow or model behavior | | Do you have labeled data at useful scale? | Fine-tune becomes plausible | Stay with prompting or retrieval | | Will this capability be core for years? | Build leverage matters | Keep complexity low |

2.3 Reversibility and one-way doors

Not all decisions deserve the same ceremony. Some choices are two-way doors. You can reverse them cheaply. Some choices are one-way doors. You reverse them only with pain, risk, and politics. A Lead must know the difference early. | Decision | Door type | Why | |---|---|---| | Prompt wording tweak | Two-way | Cheap rollback, low blast radius | | Swapping an SDK wrapper | Two-way | Mild migration cost | | Vendor model commitment in contracts | One-way-ish | Legal and integration inertia | | Data retention policy | One-way-ish | Compliance and trust implications | | Training your own model stack | One-way | Team topology and infra commitment |

Reversible decisions should move quickly. Irreversible decisions deserve deeper weather checks. This principle protects both speed and caution. Without it, teams either rush everything or over-analyze everything. Both are costly. Decision quality is partly about matching process to reversibility. 

High reversibility + low blast radius -> decide fast
High reversibility + high blast radius -> add canary and monitor
Low reversibility + low blast radius -> document and gate carefully
Low reversibility + high blast radius -> escalate, prototype, and ADR

2.4 Decision records and ADRs

Good captains keep a ship's log. Good Leads keep ADRs. An ADR is not bureaucracy. It is memory made searchable. When people leave, ADRs keep context alive. When incidents happen, ADRs reveal previous tradeoffs. When onboarding begins, ADRs reduce folklore. That is why ADRs compound. 

ADR template:
- Title
- Status
- Context
- Options considered
- Decision
- Consequences
- Triggers to revisit

A weak ADR says, "We chose X." A strong ADR says why alternatives lost. It also says what evidence would change the decision. That final piece is important. Principles without revisit triggers become dogma. A living ship's log beats a holy book. | ADR field | What it protects against | |---|---| | Context | Memory loss about constraints | | Options considered | False myth that no alternatives existed | | Decision | Ambiguity in ownership | | Consequences | Surprise costs later | | Triggers to revisit | Frozen thinking when reality changes |

Write ADRs when cost or coupling is non-trivial. Do not write them for every tiny prompt tweak. Again, match process to reversibility.

2.5 Cost of wrong decisions across stages

Wrong decisions hurt differently at different stages. At prototype stage, speed matters most. At pre-production stage, correctness and learnings matter more. At production stage, reliability and operability dominate. Many teams use the same decision rule for all stages. That is a category mistake. | Stage | Best default bias | Wrong decision usually costs | |---|---|---| | Prototype | Learn fast | Some rework, low external damage | | Pilot | Validate with guardrails | Reputational dent, trust loss | | Production | Reliable repeatability | Incidents, toil, legal exposure | | Platform | Long-term leverage | Multi-team slowdown for months |

Using heavy process too early slows exploration. Using light process too late breaks production. The art is knowing when the stage changed. Leads watch for that stage shift explicitly. They say, "This is no longer a demo system." Then they tighten the compass.

Scenario: customer support assistant

Suppose product wants an AI support assistant in six weeks. The naive team debates model brands on day one. The principled team writes decision questions first. Can we tolerate occasional wrong answers? Will answers cite company policy? Do agents need approval before customer-facing actions? What happens during provider downtime? Those answers narrow the design quickly. Maybe the first version uses an API plus retrieval. Maybe escalation stays human for refunds. Maybe an ADR records why self-hosting was deferred. That is not less engineering. That is more adult engineering.

Chapter 3: Code and system quality

AI systems create a special confusion around quality. Teams often ask, "How do I test a probabilistic system?" The answer is not to avoid testing. The answer is to test the right layers differently. You do not test everything with one instrument. A captain does not use the same tool for wind and engine oil.

3.1 What to unit test

Unit tests still matter in AI systems. They just target the deterministic parts. | Deterministic component | Example unit test | |---|---| | Prompt builder | Required variables always inserted | | Retrieval filter | Tenant isolation never breaks | | Post-processing parser | JSON schema violations handled cleanly | | Tool router | Wrong tool cannot execute without permission | | Cost calculator | Token accounting remains correct | | Fallback logic | Timeout triggers known backup path |

Unit tests protect contracts around the model. They do not prove model quality by themselves. But they stop silly breakages from pretending to be model failures. That separation saves debugging time. When deterministic glue is shaky, every incident feels mysterious. Strong Leads remove fake mystery first. Then they inspect the model layer.

3.2 What to evaluate

Evals handle the probabilistic part. They ask whether behavior is good enough for the real task. Different systems need different eval sets. A support assistant, coding agent, and medical summarizer need different rubrics. | Eval target | Example question | |---|---| | Accuracy | Did the answer match ground truth? | | Faithfulness | Did the answer stay inside retrieved evidence? | | Safety | Did the system avoid harmful or disallowed actions? | | Helpfulness | Was the response actually useful to users? | | Latency-quality balance | Did faster routing hurt quality too much? | | Cost-quality balance | Are expensive tokens buying meaningful improvement? |

A simple rule helps here. Unit test the wiring. Eval the behavior. Monitor the operation. These three layers together create quality discipline. 

Quality stack
├─ Unit tests -> deterministic correctness
├─ Evals -> task behavior
└─ Monitoring -> live production health

Many teams jump straight to live monitoring. That means production becomes the first eval suite. Customers become unpaid testers. That is unacceptable for serious systems.

3.3 Observability and error budgets

Observability means you can explain system behavior after the fact. You log inputs, outputs, traces, latency, cost, model versions, and failure classes. Without those signals, incidents become storytelling contests. With those signals, incidents become engineering. | Signal | Why it matters | |---|---| | Request trace | Reconstructs the full chain of execution | | Model version | Catches silent vendor drift | | Retrieval sources | Explains hallucination or empty context | | Latency buckets | Distinguishes slow model from slow tools | | Cost per request | Prevents invisible budget explosions | | User feedback or judge score | Provides online quality signal |

Error budgets add operating discipline. They answer a practical question. How much unreliability can we tolerate before work must shift? If the system keeps burning the budget, feature work pauses. Reliability work becomes mandatory, not optional. This protects the crew from optimistic denial. It also protects product from vague reliability arguments. | Reliability state | Team behavior | |---|---| | Budget healthy | Ship features normally | | Budget thinning | Increase caution, tighten rollout | | Budget exhausted | Freeze risky launches, fix operations |

The exact numbers are contextual. The principle is not. You need a forcing function for reliability tradeoffs.

3.4 Technical debt in AI systems

Technical debt in AI systems is not just messy code. It also lives in prompts, datasets, evals, workflows, and human handoffs. | Debt type | Example | Usual future pain | |---|---|---| | Prompt debt | Long fragile prompt patched by many people | Hidden regressions | | Data debt | Unclear source quality or labeling | Fine-tune confusion | | Eval debt | Tiny or stale benchmark set | False confidence | | Workflow debt | Agent loop with unclear stop conditions | Cost spikes and incidents | | Ops debt | No rollback or emergency disable | Long incidents | | Knowledge debt | Context lives in one person's head | Slow onboarding |

AI debt is sneaky because demos still work for a while. The bill arrives later under scale, turnover, or failure. A Lead must see delayed costs early. That is part of the captain role.

Scenario: the Friday regression

Picture a Friday incident. Answer quality suddenly collapses for one customer segment. A weak team blames the model immediately. A strong team checks the layers in order. Did retrieval sources change? Did a prompt template change? Did a vendor model version update silently? Did latency fallback route more traffic to a weaker model? This ordered investigation is itself an engineering principle. It keeps panic from outrunning evidence. 

Weather check during incident
├─ Is blast radius high?
│  ├─ Yes -> rollback or degrade safely first
│  └─ No  -> keep tracing while containing impact
├─ Do we know what changed?
│  ├─ Yes -> test the suspected change quickly
│  └─ No  -> inspect versions, prompts, retrieval, and traffic patterns
└─ Is customer harm possible?
   ├─ Yes -> add HITL or stop action path
   └─ No  -> continue controlled diagnosis

Chapter 4: Team and process

Principles become durable only when team process reflects them. A brilliant Lead with weak process becomes a bottleneck. A solid Lead with healthy process creates repeatable quality.

4.1 Code review culture for AI code

AI code review must go beyond syntax and style. It must ask whether the system behavior is reviewable at all. | Review question | Why it matters | |---|---| | What changed in prompts, tools, or retrieval? | Hidden behavioral shifts are common | | Which evals cover this change? | Prevents review by vibes | | What is the fallback path? | Protects users during failure | | What are the new failure modes? | AI changes expand risk surface | | Is cost or latency affected? | Product impact is not only quality |

Review culture should reward clarity, not cleverness. A prompt diff without context is not reviewable. A model switch without an eval report is not reviewable. A tool call path without guardrails is not reviewable. These are cultural standards, not optional niceties.

4.2 Documentation standards

Documentation should be light enough to survive, yet strong enough to guide. You do not need ten documents for one feature. You do need the right five documents for the system. | Document | Minimum useful content | |---|---| | README | System purpose, setup, architecture, owners | | ADR | Major technical choices and revisit triggers | | Eval spec | Tasks, datasets, metrics, thresholds | | Runbook | Incident steps, rollback path, contacts | | Prompt card or model card | Version, purpose, limits, known risks |

These documents form the ship's log. Without them, the crew argues from memory. Memory is a terrible production system.

4.3 Knowledge sharing and mentoring

Strong teams reduce hero dependency deliberately. That means demos, design walkthroughs, pairing, and postmortems. Knowledge sharing is not ceremony after the real work. It is the real work for team durability. Mentoring also matters here. Junior engineers usually copy visible behavior. If you only model speed, they learn recklessness. If you model principled speed, they learn judgment. Leads teach through repeated decisions, not speeches.

4.4 Sprint planning for research-heavy work

Research-heavy AI work breaks normal sprint planning if treated like deterministic feature work. You cannot promise exact output when discovery is still happening. But you also cannot hide behind permanent ambiguity. The middle path is outcome-based planning. | Work type | Planning style | |---|---| | Exploration spike | Time-boxed question with exit criteria | | Prototype path | Demo plus measured eval delta | | Hardening path | Reliability and operations milestones | | Launch path | Rollout plan, guardrails, owner handoff |

Plan research tasks around questions, not fantasy certainty. For example, ask whether retrieval lifts accuracy by ten points. Do not ask for a fully solved agent by Friday. That is how teams stay honest. 

Healthy research sprint board
To learn -> To validate -> To harden -> To ship

When ambiguity is high, shorten feedback loops. When risk rises, increase documentation and demos. That is the process version of weather checking. It keeps the crew aligned even when answers are incomplete.

Scenario: automate or keep manual

Module 17 assumes you know when to automate. That choice starts here. | Question | Lean manual when yes | Lean automate when yes | |---|---|---| | Is volume low? | Yes | No | | Is risk high and edge cases unclear? | Yes | No | | Is the task repeated and stable? | No | Yes | | Can we observe failures cheaply? | No | Yes | | Is response time critical? | No | Yes |

Manual-first is not anti-engineering. It is often good risk management. Automation should arrive when stability, volume, and observability justify it.

Chapter 5: Communication and influence

A Lead who cannot explain tradeoffs will lose good decisions politically. Communication is not decoration on top of engineering. It is part of how engineering gets adopted.

5.1 Translating decisions for stakeholders

Different stakeholders hear different risks. The same decision must be translated without distortion. | Audience | What they care about | How to translate | |---|---|---| | Product manager | User value, timeline, scope | Explain capability and confidence bands | | Legal or compliance | Data use, auditability, controls | Explain retention, approvals, logging | | Finance | Run cost and scaling curve | Explain variable cost and break-even | | Executive | Business leverage and risk posture | Explain options, upside, and exposure | | Support or ops | Failure handling | Explain fallback, escalation, and ownership |

Good translation does not hide uncertainty. It packages uncertainty in business language. For example, do not say only, "The eval is noisy." Say, "We can promise a safe pilot, not broad automation yet." That keeps trust intact.

5.2 Writing RFCs and presenting tradeoffs

RFCs are pre-decision communication tools. They create shared understanding before implementation hardens. A good RFC frames the problem, options, recommendation, and non-goals. It also names risks openly. Stakeholders trust engineers who surface tradeoffs early. | Tradeoff pair | Typical question | |---|---| | Speed vs reliability | Do we launch now or add one more gate? | | Cost vs quality | Are premium tokens buying enough value? | | Control vs simplicity | Should we self-host or use a managed API? | | Automation vs oversight | Can this action stay human-approved? | | Flexibility vs standardization | Should teams choose freely or use a platform default? |

Present tradeoffs as decisions, not as complaints. Then recommend a default path. Leadership is not listing all possibilities forever. Leadership is narrowing choices responsibly.

5.3 Managing expectations for AI capabilities

AI systems invite magical thinking from stakeholders. Leads must defuse that politely and repeatedly. Do not promise deterministic perfection from probabilistic systems. Do not promise full automation when the workflow still needs humans. Do not promise future fine-tuning miracles without data evidence. Expectation management is not negativity. It is trust-preserving honesty.

Scenario: the executive demo request

Suppose an executive asks for a polished demo by Friday. You can say yes recklessly. Or you can say yes with boundaries. Maybe the demo will use curated data only. Maybe customer actions remain disabled. Maybe the narrative clearly separates prototype from launch readiness. That answer protects momentum without creating future betrayal.

5.4 Retrieval prompts

  1. Explain build vs buy vs fine-tune using reversibility, learning speed, and risk.
  2. Describe the difference between unit tests, evals, and monitoring in AI systems.
  3. Give an example of a one-way door decision in AI architecture and how you would document it.
  4. How would you plan a sprint when half the work is research and half is hardening?
  5. Translate an AI technical decision differently for product, legal, and finance.

5.5 Honest admission

Principles are context-dependent. There are no universal rules that fit every team, market, and risk posture. API first is a strong default, not a religion. Manual review is often wise, not always necessary forever. More documentation is not always better documentation. Good Leads hold principles firmly and ego lightly. They revisit the compass when the sea genuinely changes.

Chapter 6: Recap and application

6.1 Failure-fix table

Failure Principle-based fix
Fast launch, no memory Write ADRs and maintain the ship's log
Repeated architecture debates Use explicit decision criteria and RFCs
Model blamed for every issue Separate unit tests, evals, and monitoring
Friday regressions Add versioned prompts, traces, and rollback paths
Expensive experimentation Bias toward reversible, low-ops defaults
Onboarding takes weeks Standardize docs, runbooks, and walkthroughs
Research work misses every sprint Plan around questions and exit criteria
Stakeholders expect magic Communicate capability bands and risk openly
Unsafe automation Keep humans in the loop until confidence rises
MLOps feels premature or overwhelming Automate only when volume, risk, and repetition justify it

6.2 Key points to remember

Principles are decision tools under uncertainty. They matter most when pressure is high and information is partial. Start with the decision, not the fashionable tool. Prefer reversible paths early. Document high-coupling choices with ADRs. Unit test deterministic glue. Eval model behavior on real tasks. Monitor live systems because behavior drifts. Plan research work around questions, not wishful deadlines. Translate technical tradeoffs into stakeholder language. Good leadership narrows choices responsibly.

6.3 Interview questions

  1. Your team wants to fine-tune immediately because quality feels poor. How would you respond?
  2. Tell me about a one-way door AI architecture decision and how you documented it.
  3. A prototype is now customer-facing. What principles change when moving into production?
  4. How do you review prompt or agent changes with the same rigor as code changes?
  5. How do you plan sprint commitments when half the work is exploratory research?
  6. A stakeholder wants full automation for a high-risk workflow. What is your response?
  7. What would you log and monitor first for an LLM feature already in production?

These questions test judgment more than vocabulary. Answer with context, tradeoffs, decision process, and outcome. That pattern signals Lead-level thinking.

6.4 Production experience signals

Interviewers trust stories with operating details. Bring examples of incidents, rollbacks, tradeoff memos, and scope decisions. Mention where you chose manual review deliberately. Mention where you delayed automation because observability was weak. Mention where ADRs or RFCs prevented repeated debate. Principles feel real when attached to scars.

6.5 Exercises

  1. Pick one AI feature you know well. Write a build-vs-buy memo in one page.
  2. Draft an ADR for model choice, including triggers to revisit the decision.
  3. Split one existing system into unit tests, evals, and monitoring signals.
  4. Create a risk matrix for one workflow with user harm, cost, and compliance axes.
  5. Run a mock design review where you translate the same choice for product and finance.
  6. Rewrite a vague research sprint into time-boxed questions with exit criteria.
  7. Identify three kinds of technical debt in one AI project you have seen.
  8. Decide where manual review should remain in that project for the next quarter.

6.6 Foundation-gap audit for Module 17

Module 17 assumes four habits are already in place. | Assumed habit | What "ready" looks like | |---|---| | Decision framework basics | You can explain a major architecture choice with criteria and tradeoffs | | When to automate vs manual | You can defend a staged path from human review to automation | | Risk assessment | You can evaluate blast radius before rollout or tool execution | | Documentation habits | You write ADRs, runbooks, and evaluation notes without being chased |

If these habits are weak, MLOps will feel like random infrastructure. If these habits are strong, MLOps will feel like principled automation.

6.7 Bridge to the next module

Next module — 04_ml_platform_operations — operationalizes these principles into concrete infrastructure: CI/CD for ML, model registries, monitoring, and the platform that makes good engineering automatic. Today you built the compass. Next, you will build the machinery that enforces it. That is the transition from judgment to systems.