Skip to content

04. Week 1 — Daily Recall

Spaced practice. Answer from memory. If stuck on a question, jump to the explainer chapter referenced in parens.

Monday (after explainer chapters 1-2)

  1. Why does one perceptron fail at XOR? Walk the geometric reason. (§1.1)
  2. Universal approximation — what does it guarantee, what doesn't it tell you? (§2.2)
  3. Why do two stacked linear layers collapse to one? (§2.1)
  4. Write the forward pass for a 2-layer MLP with ReLU. (§2.2)
  5. The hand-picked XOR-solving network uses y = h₁ − 2·h₂. What does each h represent in plain language? (§2.2)

Tuesday (after explainer chapter 3)

  1. He init: what's the variance formula? Why does ReLU need 2/fan_in and not 1/fan_in? (§3.1)
  2. What goes wrong if init is too large? Too small? All zeros? (§3.1)
  3. Explain backprop in one sentence using the chain rule. (§3.2)
  4. Difference between an iteration, a step, and an epoch? (§3.3)
  5. Why does mini-batch noise help generalization? (§3.3)
  6. Why is the cross-entropy + softmax gradient just p − y? Why is that clean? (§3.4)
  7. What does softmax guarantee about its output? (§3.4)

Wednesday (after explainer chapter 4)

  1. Sigmoid's max derivative is 0.25. Why does that cause vanishing gradient through 10 layers? (§4.1)
  2. ReLU's "dead neuron" problem — what causes it, how do you detect it in production? (§4.1, §6.4)
  3. Why does the canyon-shaped loss surface trip plain SGD? (§4.2)
  4. AdamW vs Adam — what does decoupled weight decay actually fix? (§4.2)
  5. What is learning-rate warmup, and why do transformers need it? (§6.3)

Thursday (after explainer chapter 5)

  1. Why is a power law a straight line on log-log paper? (§5.1)
  2. Chinchilla's key finding in one sentence. (§5.1)
  3. Llama 3 trained an 8B model on 15T tokens (~1875:1 ratio). Why ignore Chinchilla? (§5.1)
  4. Three signs you are overfitting. Three fixes. (§5.2)
  5. Dropout — what does it do mechanically and why does that help? (§5.2)
  6. What is double descent? Why is it a puzzle? (§5.2)

Friday (cumulative)

  1. From memory: draw the full pipeline (input → init → forward → loss → backward → update).
  2. Self-supervised vs supervised — give one example of each. Which one trains LLMs? (§9 of study_material)
  3. Three things we still do not fully understand about deep networks. (§5.3)

Weekend (pre-hands_on_lab)

  1. Shapes of W, x, b in a layer with 784 inputs and 128 outputs?
  2. Why does MNIST use softmax + cross-entropy and not sigmoid + MSE? (§3.4)
  3. List all 11 failures and 11 fixes from explainer §6.1 — sketch the table from memory.
  4. For each row in the table, name the corresponding ELI5 placeholder (rule pile, bend, nudge, etc.) where it applies.