14. Honest admission — the clean story is not the whole story¶

~8 min read. The simple decoder story is correct, but production systems add many efficiency tricks.

Built on the ELI5 in 00-eli5.md.

The exam rule still survives in production.

The implementation just gets less tidy.

Good engineers keep the simple model and the messy reality together.

1) What this module simplified on purpose¶

This module taught the decoder in a clean order.

That was useful, and it was selective.

We used dense attention.

We used one K and one V per query head.

We used simple positional embeddings and straight cache growth.

Those choices are perfect for first principles.

They are not the whole production picture.

The goal was clarity first.

The exam rule had to feel obvious.

The projection lens and parallel graders had to feel concrete.

The memory shortcut had to feel real.

Once those ideas stick, we can add the mess.

That is what this file does.

2) Real systems change the attention kernel and head layout¶

The teaching code materialized the full attention matrix.

That is the naive O(n^2) memory story.

Production kernels often avoid that writeback.

FlashAttention is the standard example.

It tiles the computation and keeps more work on chip.

That reduces memory traffic sharply.

That is why PyTorch and vendor stacks prefer fused attention paths.

We skipped that because kernel design is a different lesson.

Real systems also change head sharing.

The clean story gave every query head its own K and V.

Many modern models do not.

Grouped-query attention shares K and V across groups of query heads.

Multi-query attention shares one K head and one V head across all queries.

That shrinks the KV cache.

It matters because decode is usually memory-bound.

┌─ MH ───────────┬─ GQA ──────────┬─ MQA ──────────┐
│ q0──→k0,v0     │ q0──→k0,v0     │ q0──→k0,v0     │
│ q1──→k1,v1     │ q1──→k0,v0     │ q1──→k0,v0     │
│ q2,q3──→own KV │ q2,q3──→k1,v1  │ q2,q3──→k0,v0  │
└─ 4 KV (1:1) ───┴─ 2 KV (2:1) ───┴─ 1 KV (4:1) ───┘

The query heads still exist.

The KV streams become fewer.

That keeps the memory shortcut smaller during decode.

3) One numerical example: why GQA saves memory¶

Take one layer with T = 4000 cached tokens.

Let d_head = 128.

Let there be 32 query heads.

Assume FP16 storage, so each number uses 2 bytes.

Store both K and V.

Naive per-head KV cache¶

Numbers stored are 2 * 32 * 4000 * 128.

32 * 128 = 4096.

4096 * 4000 = 16,384,000.

2 * 16,384,000 = 32,768,000 numbers.

32,768,000 * 2 = 65,536,000 bytes.

That is about 62.5 MB.

GQA with 8 KV heads¶

Numbers stored are 2 * 8 * 4000 * 128.

8 * 128 = 1024.

1024 * 4000 = 4,096,000.

2 * 4,096,000 = 8,192,000 numbers.

8,192,000 * 2 = 16,384,000 bytes.

That is about 15.6 MB.

So the cache is about 4x smaller.

Same context length.

Much less memory traffic.

That is why serving teams care.

4) Real systems also change positions and cache management¶

We used additive positional embeddings for teaching.

Many decoder models now use RoPE instead.

RoPE rotates queries and keys by position-dependent angles.

That makes relative position information appear naturally in dot products.

You should know the name, even without the derivation.

Cache handling also gets smarter.

Simple concatenation is easy to teach.

Large serving systems use paged KV memory.

They treat cache more like virtual memory.

Pages and shared prefixes can be reused.

That is prefix caching.

Some systems also quantize KV storage.

INT8 halves memory versus FP16.

INT4 can reduce it further.

Those tricks do not change causality.

They change cost and throughput.

5) Some patterns narrow attention, and some questions stay open¶

The teaching story used full causal history.

Some real models use sliding-window attention instead.

A token sees only a recent legal band.

That still obeys the exam rule.

It just forgets very old context during scoring.

Where this lives in the wild¶

PyTorch 2.x — scaled_dot_product_attention often dispatches to fused, memory-efficient kernels.
LLaMA-family models — grouped-query attention reduces KV storage while preserving many query heads.
vLLM serving — paged KV memory makes long-context serving practical across many requests.
Mistral-style decoders — sliding-window attention trades some history for lower compute and memory cost.
Mamba-style alternatives — state-space models keep pressure on attention-first assumptions.

Pause and recall¶

Why did this module avoid implementing FlashAttention directly?
How does GQA reduce the size of the KV cache?
What problem does paged KV memory solve that simple concatenation ignores?
Why is sliding-window attention still causal?

Interview Q&A¶

Q. What major kernel optimization did this module skip?

A. It skipped FlashAttention-style kernels and used the naive full-score-matrix story.

Common wrong answer to avoid: “Production attention just means using larger GPUs.”

Q. What is GQA, and why do serving teams like it?

A. Many query heads share fewer KV heads, so decode uses less cache memory and bandwidth.

Common wrong answer to avoid: “GQA means each head gets a larger query vector.”

Q. Why is paged KV cache important?

A. It manages cache efficiently across many requests, much like virtual memory pages.

Common wrong answer to avoid: “It mainly changes the math inside softmax.”

Apply now (5 min)¶

Redo the GQA cache calculation with 16 KV heads instead of 8.

Write each intermediate multiplication.

Then state the memory ratio versus the naive 32-KV-head design.

Next, sketch from memory:

the naive-versus-GQA-versus-MQA ASCII diagram,
one sentence on why FlashAttention reduces memory traffic,
and one sentence on what paged KV memory borrows from operating systems.

Bridge. You can now code the decoder core. How do real organizations train these blocks at scale? That is Module 05.

→ ../05_llm_training_pipeline/00-eli5.md