Skip to content

04. Explainability and citations

Uncertainty tells the user when to be careful. Explainability tells the user why the AI gave its answer. The discipline is the UX patterns that turn the AI from a black box into an inspectable system.

A platform engineer at a Mumbai legal-tech company notices users frequently asking the AI "where did you get this?" The team adds source citations to every retrieval-grounded response — the AI says X; below the response is "Source: contract page 14, section 7.3 [link]". Users now verify the AI's claims by clicking through to the source; trust in correct responses increases; the small number of cases where the AI got it wrong are caught by users who notice the source does not match the claim.

This chapter is the explainability discipline.

What explainability is

Explainability is the UX practice of surfacing the AI's sources, reasoning, or basis for its response — so users can inspect, verify, and trust appropriately.

Three forms: citations, reasoning, evidence.

Citations for retrieval-grounded responses

The most concrete and most common form. When the AI's response is based on retrieved documents:

  • The cited source appears inline or below the response.
  • A click goes to the source (or to a preview).
  • Multiple sources are clearly distinguished.
The policy effective date is 2026-01-15.
Source: Customer Agreement v2.1, Section 4.2 [view document]

The citation is specific (which document, which section); the user can verify.

Citations require the AI's response to be traceable to its sources. The retrieval pipeline must capture which sources contributed to the response; the response must be structured to attach citations correctly.

Reasoning for inferences and recommendations

For responses that involve reasoning beyond retrieval:

  • Show the major steps of the reasoning briefly.
  • Highlight the inputs that informed the conclusion.
  • Let the user expand for more detail.
We recommend Plan B for your team.

Why: You mentioned needing real-time support (Plan A doesn't include this)
and your team size is 15+ users (Plan A is for ≤10). Plan B fits both.

[Show full analysis]

The reasoning is bullet-point summary; the full analysis is available on demand.

Evidence for predictions and assessments

For responses that include a judgement (an estimate, a prediction, a risk assessment):

  • The evidence base is summarised: "Based on these 5 similar cases..."
  • The confidence is surfaced (chapter 03).
  • The user can drill into the evidence.
Estimated time to resolution: 2-3 business days.

Based on:
- 12 similar cases in the past 30 days (avg 2.4 days)
- Current queue depth (3 ahead)
- Your account tier (premium = faster routing)

[View case details]

The evidence is specific; the user can challenge or verify.

What to cite, what to skip

Not every response needs explicit citation. Cite when:

  • The response includes specific factual claims (dates, numbers, names).
  • The cost of being wrong is high.
  • The user needs to verify or take action based on the response.
  • The user is in a workflow where audit or accountability matters.

Skip when:

  • The response is casual conversation.
  • The response is creative or generative (not factual).
  • The citation would be uninformative ("this is general knowledge").

The discipline is judgement; over-citation produces noise; under-citation leaves users guessing.

Citation quality

Citations are only useful if they hold up under inspection.

  • The source actually says what the AI claims.
  • The link works.
  • The source is accessible to the user (not behind a permission wall).
  • The source is current (not citing a deprecated document).

A platform with broken citations damages trust faster than no citations. The citation infrastructure must be reliable.

Reasoning without overwhelming

The reasoning UX has to balance "enough to be useful" with "not too much to read."

A good pattern: short summary by default (3-4 bullet points); expandable full detail on click. Users who want depth get it; users who want a summary are not overwhelmed.

A bad pattern: 200-word reasoning block under every response; users skip past it; the signal is lost in volume.

Chain-of-thought visibility

For complex reasoning, showing the AI's chain of thought (the intermediate steps) is sometimes useful:

[User asks: should we extend our trial customer's trial?]

Thinking:
1. Checking trial start date... 14 days ago.
2. Checking usage pattern... high engagement; daily logins.
3. Checking conversion rate for similar profiles... 65%.
4. Cost vs benefit... low cost; high upside.

Recommendation: Extend by 7 days.

The chain shows the AI considered the relevant factors; the user can trust the recommendation more.

When to use: complex multi-step reasoning where the user might otherwise wonder "did the AI consider X?"

When to skip: simple lookups; the chain would be overhead.

Common mistakes

No citations on retrieval-grounded responses. Users cannot verify; trust loss on errors.

Broken citations. Links to deleted documents; sources that don't say what cited.

Over-citing. Every routine response loaded with sources; signal lost.

Reasoning without summary. Wall of text; users skip.

Reasoning that does not match the actual logic. The AI generates plausible-sounding reasoning that does not reflect what produced the response.

Interview Q&A

Q1. When should responses include citations? When the response includes specific factual claims; when the cost of being wrong is high; when the user needs to verify or act; when audit matters. Skip casual or generative responses. The judgement is per response type; over-citation is noise, under-citation leaves verification impossible. Wrong-answer notes: "always" produces fatigue; "never" produces unverifiable responses.

Q2. What makes a good citation UX? The source is specific (document name, section, page if relevant). A click goes to the source or a preview. The source is accessible to the user. The source is current. The citation infrastructure is reliable (no broken links). When users follow the citation, the source actually says what was claimed. Citations that fail any of these damage trust more than no citations. Wrong-answer notes: vague citations (just a document name) are not actionable.

Q3. Walk through the reasoning-UX trade-off. Reasoning helps users understand the AI's basis for its response; it is necessary for complex inferences. Reasoning at length under every response produces noise users skip. The balance: short summary (3-4 bullets) by default; expandable full detail on click. Users who want depth get it; users who want the answer get it without wading through reasoning. Match the depth to the complexity of the inference. Wrong-answer notes: uniform reasoning depth across all responses is wrong.

Q4. The AI's generated reasoning does not match the actual computation. What is the concern? Honesty. Generated reasoning that is post-hoc rationalisation (the AI describes "why" after the fact, not the actual factors) misleads users; they trust a reasoning that does not reflect the AI's actual basis. The discipline: either the reasoning is structurally tied to the actual computation (the AI computed against documented criteria, surfaced those criteria), or the response does not claim to show reasoning. Generated reasoning that is not faithful is worse than no reasoning. Wrong-answer notes: "users like seeing reasoning" misses that fake reasoning is dishonest.

What to do differently after reading this

  • Cite retrieval-grounded responses; make citations specific and reliable.
  • Summarise reasoning briefly; expandable detail for depth.
  • Show evidence for predictions and recommendations.
  • Reserve chain-of-thought for genuinely complex reasoning.
  • Avoid post-hoc generated reasoning that does not reflect actual computation.

Bridge. Explainability builds trust through transparency. The next chapter handles what happens when the AI is wrong — error and recovery flows. → 05-error-and-recovery-flows.md