05. Chain-of-thought — when step-by-step helps, and when it hurts¶
~14 min read. Reasoning prompts can rescue hard tasks, but they can also add latency, noise, and false confidence.
Built on the ELI5 in 00-eli5.md. The Work order — the task sheet for this job — sometimes needs a reasoning scaffold, not just a final answer request.
What chain-of-thought is really doing¶
See. When we ask for step-by-step reasoning, we are not adding intelligence by magic. We are changing the task shape. We are telling the model to unfold intermediate steps before the final answer. That can help on tasks with hidden subproblems. Math. Multi-rule policy checks. Planning. Structured extraction with ambiguity.
Picture first.
direct answer prompt reasoning prompt
┌──────────────────────┐ ┌─────────────────────────┐
│ read task │ │ read task │
│ jump to answer │ │ unpack steps │
└──────────┬───────────┘ │ check steps │
▼ │ then answer │
final answer └──────────┬──────────────┘
▼
final answer
Simple, no? The model already has many latent patterns. A reasoning prompt can make it traverse them more carefully. The Work order becomes a sequence request instead of a pure endpoint request. That can reduce errors when the task genuinely decomposes.
Now what is the problem? People hear "chain-of-thought" and apply it everywhere. Bad idea. If the task is easy, extra reasoning just adds tokens. If the reasoning path is wrong, the model may confidently walk deeper into error. If you expose long reasoning in a product reply, you may leak noise, latency, or unwanted internal detail.
When reasoning prompts help most¶
They help when the task has several dependent checks. For example, refund eligibility may depend on plan type, renewal window, usage threshold, and approval path. A direct answer can skip one condition. A stepwise prompt can force the model to inspect each gate.
question
│
▼
┌─────────────┐
│ check gate 1│ plan type
├─────────────┤
│ check gate 2│ time window
├─────────────┤
│ check gate 3│ usage threshold
├─────────────┤
│ check gate 4│ approver
└──────┬──────┘
▼
final answer
They also help when you want explicit audit traces. Maybe the user does not see the full trace, but your workflow logs it. That is useful in internal tools. Analyst copilots. Operations triage. Back-office QA. In such cases, reasoning is not decoration. It is an intermediate artifact.
There are gentler versions too. You do not always need, "Show all reasoning." Often, "Check the policy in three steps before answering" or "List the criteria you used" is enough. A compact checklist can give most of the gain with less noise. That is often better for production UX.
When reasoning prompts hurt¶
Look. They hurt when the task is simple and deterministic. If the model only needs to classify one label, long reasoning may introduce drift. They hurt when users need speed. More tokens mean more latency and cost. They hurt when the model invents elegant but false logic. A wrong chain can feel more persuasive than a wrong short answer. That is dangerous.
They also hurt when you confuse reasoning with truth. A detailed explanation is not proof. The model can rationalize after the fact. So what to do? Evaluate final correctness, not just explanation quality. And whenever possible, anchor reasoning to retrieved evidence or tools.
Here is the contrast.
helpful use harmful use
┌──────────────────────┐ ┌────────────────────────┐
│ multi-step check │ │ simple label task │
│ hidden criteria │ │ forced long reasoning │
│ audit value │ │ extra latency │
└──────────┬───────────┘ └──────────┬─────────────┘
▼ ▼
fewer skipped steps more noise, same task
The Standing rulebook should define whether visible reasoning is allowed. The Reply form should define what the user sees. Maybe internal traces stay internal. Maybe the customer sees only a short rationale. That split matters.
Worked example — direct answer vs guided reasoning¶
Suppose an internal support assistant must decide refund eligibility. Policy summary: Enterprise annual plans can request a refund within 30 days of renewal. No refunds after 5,000 API calls in the renewal term. Support verifies. Billing operations approve.
User case: Annual plan. Renewed 18 days ago. Usage since renewal: 6,200 API calls. User asks, "Am I eligible for a refund?"
Direct prompt first.
Possible model response.
See the skipped step? The model noticed the time window. It missed the usage threshold. Now try a guided reasoning prompt.
[SYSTEM]
You are a billing policy assistant.
Check refund eligibility in this order:
1. Verify plan type.
2. Verify renewal window.
3. Verify usage threshold.
4. State approver.
Then give the final answer in two bullets.
[USER CASE]
Plan: enterprise annual
Days since renewal: 18
API calls since renewal: 6200
Question: Am I eligible for a refund?
Possible model response.
- Check: The plan type and 30-day window fit the refund policy, but the account has exceeded 5,000 API calls since renewal.
- Final answer: Not eligible for a refund under the provided policy. Billing operations would be the approver if the account were eligible.
Simple, no? The reasoning scaffold prevented one skipped gate. The Work order changed from, "answer fast" to "check in order, then answer." That is often the difference.
Now one better production variant. Do not expose every internal thought. Ask for a short
checklist plus answer. For example, "Return checks_passed, checks_failed, and
final_answer." That keeps reasoning structured and auditable. It also keeps the
customer-facing output tight.
Production guidance for reasoning prompts¶
Use explicit steps when the task truly has stages. Keep the step count low. Two to five steps is often enough. Avoid grand essays. Name the checks concretely. "Verify source." "Compare dates." "Compute threshold." Much better than, "Think deeply."
Pair reasoning prompts with grounded inputs. If the task depends on policy text, include the policy. If it depends on math, show the numbers. If it depends on tools, use tools. Reasoning on missing evidence just creates polished guessing.
Finally, measure whether reasoning actually helps. On some tasks, it improves accuracy sharply. On others, it only increases cost. Production prompt design means we do not worship one trick. We test it.
Where this lives in the wild¶
- OpenAI reasoning-model workflows — product engineers often ask the model to break hard tasks into checks or tool calls, then expose only concise user-facing rationales.
- Anthropic Claude analyst tools — teams sometimes structure prompts as ordered verification steps so long-document analysis does not skip key evidence gates.
- Harvey legal review flows — legal-tech designers use issue-checklists before final drafting because missing one element matters more than writing a pretty paragraph.
- GitHub Copilot coding assistance — some internal repair or explanation prompts benefit from stepwise planning, while trivial tasks are answered directly to keep latency low.
- Enterprise finance bots on Bedrock or Azure OpenAI — operations teams use compact eligibility checklists for policy decisions instead of free-form reasoning paragraphs.
Pause and recall¶
- When does chain-of-thought prompting usually help most?
- Why can long reasoning make a product answer worse?
- What is the difference between visible reasoning and internal reasoning artifacts?
- Why is a compact checklist often better than "think deeply"?
Interview Q&A¶
Q: Why use reasoning scaffolds for some tasks and direct prompts for others? A: Reasoning scaffolds help when tasks have dependent checks or hidden substeps, but they waste tokens and may add noise on simple tasks.
Common wrong answer to avoid: "Always ask for chain-of-thought because more reasoning means more accuracy." Extra reasoning can also mean extra drift, latency, and persuasive mistakes.
Q: Why should final evaluation focus on correctness rather than explanation quality? A: A model can generate a fluent explanation for a wrong answer. The explanation is useful only if it supports verifiable task success.
Common wrong answer to avoid: "If the reasoning looks detailed, the answer is probably right." Fluency is not evidence.
Q: Why might a compact checklist outperform a long step-by-step essay in production? A: It preserves the helpful decomposition while reducing token cost, latency, and noisy surface text. It is also easier to parse and audit.
Common wrong answer to avoid: "Because users hate explanations." Users often like explanations. The real issue is controlled, useful reasoning.
Q: Why should reasoning prompts be anchored to evidence or tools? A: Because reasoning over missing or weak evidence often becomes structured hallucination. Evidence and tools constrain the reasoning path.
Common wrong answer to avoid: "A smart enough model does not need evidence if it reasons well." Strong reasoning on bad inputs still fails.
Apply now (5 min)¶
Exercise. Take one policy question with three checks. Rewrite the prompt so the model must inspect each gate in order. Then write a second version that asks only for a compact checklist and final answer. Compare which Reply form would be easier to audit.
Sketch from memory. Draw the staircase. Put question at the top. List three checks in the middle. Put final answer at the bottom. Mark where Standing rulebook decides whether the user sees full reasoning or only a short rationale.
Bridge. Reasoning can make the answer more correct. But if the output shape is still loose, your system still breaks. So next we force the reply into a machine-usable structure. → 06-structured-output.md