Skip to content

07. Semantic Memory — Extract the stable truths

~16 min read. Some patterns repeat often enough that they should become facts, not just remembered moments.

Built on the ELI5 in 00-eli5.md. The address-book — the place for durable facts — is how messy conversations become stable knowledge the agent can reuse.

1) What semantic memory means

Look at the contrast. 

raw turns ──→ "Can you keep replies short?"
raw turns ──→ "I usually code in Python."
raw turns ──→ "Use IST in examples."
                     │
                     ▼
semantic memory
├── prefers short replies
├── primary language: Python
└── timezone: IST
Semantic memory stores generalized facts. Not full events.

Not raw transcripts. It asks, "What stable truth should survive from repeated interaction?"

That truth may concern a user. It may concern a team. It may concern a project.

The address-book holds this kind of memory. The filing-cabinet may hold the sources. The diary-page may hold the events that produced the fact.

But the semantic layer stores the conclusion. Simple, no?

2) How facts get extracted

A good extractor is conservative.

It should ask: - is the statement explicit? - is it stable enough?

  • is it useful later?
  • is it safe to retain? 
    candidate statement
      │
      ├── repeated or explicit? ── no ──→ do not store
      │
      ├── stable beyond this task? ─ no ─→ keep as episode only
      │
      ├── useful for personalization? ─ yes ─→ address-book
      │
      └── sensitive or risky? ───── yes ─→ require stricter policy
    This is where many teams go wrong.

They store every preference-looking sentence. Then the address-book fills with temporary noise. "Today I need a long answer" is not durable.

"Prefers concise bullet summaries" may be durable. So what to do? Prefer explicit preferences.

Prefer repeated signals. Prefer facts with product value. Store source and confidence.

That keeps the address-book clean.

3) Semantic memory needs normalization

Two raw statements may mean the same thing. "Keep it brief."

"Please give short answers." These should land on one canonical fact. Otherwise duplicates pile up.

Look at the normalization flow. 

raw memory A ─┐
              ├──→ canonical key: response_style
raw memory B ─┘         value: concise
Normalization also helps conflict handling. Suppose last month the user liked long explanations.

Today the user says, "From now on keep replies short." The semantic store should update the value,

not keep both active blindly. Good semantic memory is therefore not only extraction. It is extraction plus canonicalization plus update policy.

4) Worked example: turning chat into reusable facts

Imagine these turns across three sessions. Session 1: "Please answer with bullet points." Session 1: "I work on fintech infrastructure."

Session 2: "Use Python examples when possible." Session 2: "My team is in IST." Session 3: "Do not over-explain."

From this we might store: - response_style = concise bullets - industry_context = fintech infrastructure

  • preferred_code_language = Python
  • timezone = IST Notice what we do not store.

We do not store every exact sentence. We do not store session-specific wording. We do not store irrelevant chit-chat.

The address-book now supports future turns. A new answer can automatically use bullet points. Examples can default to Python.

Deadlines can be expressed in IST. That is semantic memory earning its keep.

5) Failure modes

Failure one: false facts.

The extractor infers too much. Failure two: stale facts. A preference changed, but the store did not.

Failure three: duplicate facts. Three near-identical entries compete. Failure four: oversharing.

Sensitive traits get stored casually. Failure five: wrong scope. A team preference becomes a personal preference or vice versa.

Fixes are boring but important. Store scope fields. Store confidence.

Store source references. Allow fact overwrite or decay. Let users inspect and delete the address-book.

See. Semantic memory helps only when it stays humble.

Where this lives in the wild

  • ChatGPT Memory — OpenAI extracts long-lived user preferences such as tone, interests, and working context into a persistent memory layer.

  • Notion AI workspace assistants — team member can benefit from stable project context such as preferred templates, vocabulary, and writing style.

  • Salesforce Einstein Copilot — seller needs reusable account facts like region, product line, and buying committee preferences.
  • GitHub Copilot enterprise assistants — engineer can benefit from durable team conventions like preferred language, test style, or deployment environment.

- Shopify Sidekick — merchant benefits from stable store context such as catalog type, business goals, and preferred report format.

Pause and recall

  1. What makes a fact suitable for the address-book instead of the diary-page?

  2. Why is normalization necessary after extraction?

  3. In the worked example, which fact is clearly stable and which one would need caution?

  4. Why should semantic memory store scope and source references?

Interview Q&A

Q: Why store semantic memory separately from episodic memory?

A: Semantic memory answers stable truth questions. Episodic memory answers sequence and event questions. They need different update and retrieval policies.

Common wrong answer to avoid: "Because one is structured and the other is not" — both can be structured; the distinction is the kind of knowledge they store.

Q: Why should extractors be conservative rather than aggressive?

A: False durable facts poison future interactions. Over-remembering creates long-lived errors and privacy risk.

Common wrong answer to avoid: "Because storage is expensive" — the larger issue is correctness and trust, not bytes.

Q: Why normalize memories into canonical keys?

A: Canonical keys prevent duplication and make updates predictable. They also simplify deletion and retrieval logic.

Common wrong answer to avoid: "Only so developers can query faster" — speed helps, but semantic stability is the bigger win.

Q: Why attach confidence to semantic facts?

A: Not every extracted preference is equally certain. Confidence supports selective injection, review, and decay.

Common wrong answer to avoid: "Confidence is only for ML research" — it directly affects production retrieval and user trust.

Apply now (5 min)

Exercise:

Read five recent chat lines. Extract three candidate semantic facts. Reject at least one as too temporary. Add a canonical key and confidence score for the rest. Sketch from memory: Draw the funnel from raw turns to candidate facts to normalized address-book entries. Label where the cleanup-bell may reject storage.

Bridge. Stable truths are useful, yes. But products rarely need only personal facts. They need profiles with permissions, roles, and organization context too. → 08-user-profile-stores.md