When users say the assistant 'forgets', the history is either being truncated to fit the token budget or being summarized so aggressively that specifics vanish. The fix is a layered memory architecture: verbatim working memory, a compressed episodic summary, and a structured store of durable facts, all assembled under an explicit token budget.
Step-by-step approach
Working memory: Keep the most recent turns verbatim, budgeted by tokens rather than a fixed turn count.
Rolling summary: When the buffer overflows, fold the oldest turns into a running summary, instructing the model to preserve entities, numbers, decisions and open questions.
Fact extraction: After each turn, extract durable facts such as order IDs, preferences and constraints into a key-value or JSON store, overwriting old values when the user corrects something.
Episodic retrieval: Embed past turns and retrieve the ones most relevant to the current question, so a detail from turn 4 can resurface at turn 40.
Prompt assembly order: System prompt, then pinned user facts, summary, retrieved past turns, recent turns and finally the current question, each with its own budget.
Cross-session memory: Persist selected facts per user, with consent, so ongoing cases survive across sessions.
Trade-offs to mention
Extraction and summarization add LLM calls; run them asynchronously.
Summaries of summaries drift over time; periodically regenerate from source turns.
Persisted facts can become stale or wrong, so they need update and expiry rules.
Metrics to track
Memory recall test: plant a fact early, ask about it 20+ turns later • Contradiction rate • Tokens per request • Latency • 'I already told you' complaints
Illustrative example
An insurance claims assistant kept asking users to repeat their policy and claim numbers. Adding a small fact store for claim ID, policy number and incident date, injected at the top of every prompt, lifted the planted-fact recall test from about 60% to above 95%.
Answering the follow-ups
▸ Would you summarize older turns?
Yes, for older turns, but summarize hierarchically and tell the model explicitly to retain names, numbers and decisions. Validate summaries by checking that extracted entities survive the compression.
▸ When does persistent memory make sense?
When information has value beyond one session: preferences, ongoing cases, long-running projects. It requires consent, expiry rules and a way for users to see and edit what is stored.
▸ How would you choose which context to include?
Score candidate context by recency, relevance to the current query and importance, then fill the token budget in priority order. Pinned facts are always included regardless of score.
▸ How would memory retrieval work?
Hybrid: direct key lookup for structured facts, plus vector search over episodic memories filtered by user and session, followed by reranking and de-duplication.
0 Comments