Agent Memory: save only what is useful, permitted, and removable¶
← Back to Stage 6: RAG and Memory
Agent Memory is like a notebook with management rules. It is not a secret archive of every chat. It stores only information needed later, permitted by the user, and available to view, change, and delete.
📌 Learning goals¶
By the end of this page, you can:
- Distinguish chat history, context, RAG, and Memory.
- Distinguish short-term from long-term memory, and semantic, episodic, and procedural memory.
- Draw a memory’s lifecycle from writing and search through update and deletion.
- Set an owner, source, retention period, and deletion method for every memory.
- Use fixed tests to check that needed memories are found and unneeded ones do not remain.
🧩 Separate these four things first¶
| Core term | Plain-language picture | Precise meaning |
|---|---|---|
| Chat History | A transcript of this conversation | A message record; it does not mean every message should be saved permanently or placed in the model context. |
| Context | The material on the desk right now | The instructions, messages, tool results, and retrieved content the model actually sees for this call. |
| RAG | Go to the bookshelf when there is a question | Retrieve evidence from an external knowledge source, then give it to the model to answer. |
| Memory | A short note the assistant leaves for next time | State that must be read again across steps, threads, or sessions, with rules for writing and governance. |
The key decision: put product manuals in a knowledge base; put the current task’s progress in short-term state; only preferences saved with user consent may become long-term memory.
📚 Required reading¶
- LangChain: Memory overview — understand thread-scoped short-term memory, cross-session long-term memory, and semantic, episodic, and procedural types.
- LangGraph: Add and manage memory — see the implementation boundaries of a checkpointer, store, namespace, and semantic search.
- CoALA paper — use one shared framework to understand memory structures and operations for language agents.
- Generative Agents paper — study the classic design for recency, importance, relevance, and reflection.
- Mem0 or Letta Code — choose one current implementation and observe how it stores and retrieves state. The Letta project entry is now a landing page; current source and the App Server live in Letta Code.
⏱ Two time ranges¶
- Short-term Memory serves one thread or current task, such as messages, uploaded files, tool results, and task progress. LangGraph commonly keeps it in thread-scoped state through a checkpointer.
- Long-term Memory is needed across threads or sessions, such as user-approved preferences, project facts, or reusable experience. It must isolate users and applications with namespaces.
Short-term does not mean “only in RAM,” and long-term does not mean “never delete.” The difference is retrieval scope and lifecycle, not the name of a storage device.
🧠 Three content types¶
| Type | What it stores | Example | Risk |
|---|---|---|---|
| Semantic Memory | Relatively stable facts | The user prefers short answers; the project uses Python 3.13 | Facts can expire or conflict |
| Episodic Memory | Events and outcomes | Where the last deployment failed and which fix worked | One success does not mean it will always work |
| Procedural Memory | Rules and steps for doing work | Which gates to run before a release | Malicious content can poison future behavior |
Semantic memory and semantic search are not the same thing: the first is a type of stored content; the second is a retrieval method based on similar meaning.
🔄 A Memory lifecycle¶
- Propose a write: decide whether it really needs to be used across sessions.
- Get consent: for sensitive data or personal preferences, tell the user why it will be saved.
- Normalize: save a short fact rather than treating a whole conversation as memory.
- Add metadata: at minimum, owner, source, created_at, updated_at, expires_at, and sensitivity.
- Store in isolation: separate user/workspace/agent namespaces and apply permissions before search.
- Search and use: retrieve only a small set relevant to the current task and retain the source.
- Update or resolve conflicts: new information must not quietly coexist with old information; mark versions or replacement relationships.
- Delete and forget: users can view, change, and delete; expired data is cleared automatically, and backups need a handling policy too.
🧱 Choose the simplest design first¶
| Problem | Start with | Upgrade when |
|---|---|---|
| Fixed fields such as language, time zone, or notification preference | A direct state table | Field types grow or fuzzy search is needed |
| Freer content such as short summaries or reusable experience | Searchable text memory | Relationships, time, and conflicts become the main problem |
| People, events, and relationships change over time | Temporal Knowledge Graph | Tests show an ordinary table/search is insufficient |
| You only need to restore one workflow | Checkpoint/thread state | You truly need sharing across threads |
Start with a data table. Content that fits clear fields does not need vector search first; a problem that short-term state solves does not need permanent memory.
🛡️ Memory safety floor¶
- Do not save passwords, API keys, payment details, medical secrets, or unconsented personal data by default.
- Do not let users, tenants, workspaces, or agents share an unisolated namespace.
- Check permissions before retrieval; do not fetch a secret first and then prompt the model not to reveal it.
- Memory content is untrusted input. Validate schema, source, and prompt-injection risks before writing.
- Every memory must answer who wrote it, where it came from, when it changed, and when it will be deleted.
- Deletion must cover primary storage, search indexes, caches, and policy-managed backups.
🛠 A minimal Memory exercise¶
Save one non-sensitive preference only, such as “give the short version first.”
- Write the preference with
user_id, source, time, and retention period. - Search and read it from another thread.
- Change it to “show a table first” and confirm the old value is no longer used.
- Delete it, then search again; the result must be empty.
- Query with a different
user_id; it must not see the first user’s content.
Done when: tests for add, search, update, delete, and user isolation all pass. add alone is not enough.
Hot path, background writes, and conflicts
- Hot-path write: write immediately before answering. The result is current, but latency increases and errors affect the user directly.
- Background write: organize asynchronously after a reply. Interaction is faster, but you must handle failure, retries, and late updates.
- When one fact has newer and older versions, save time, source, and valid scope; do not randomly select one based only on vector similarity.
- Put “memory suggested by the model” in a review area first, then let rules or a user approve it; this suits high-risk content.
Common failures and a debugging order
- Nothing found: check the namespace, permission, filter, and whether storage succeeded.
- Old data found: check whether updates left conflicting versions and whether the cache refreshed.
- Too much stored: raise the write threshold and shorten retention; do not only expand the context window.
- Wrong memory: retain source and confidence, let users correct it, and never treat model inference as fact.
- Incomplete deletion: trace deletion through the primary store, index, cache, event stream, and backups.
TencentDB reading note (this entry checked: 2026-10-16 UTC): Chat Memory extracts facts and experience; Skill stores reusable procedures; LLM-Wiki organizes documents; CodeGraph indexes code relationships. Wiki and CodeGraph are knowledge-retrieval assets, not necessarily personal memory. Complete the minimal exercise first. Reviewed README / MIT license.
🎯 Curated projects and learning resources¶
Ratings represent educational value for this learning map, not a project-quality leaderboard. Choose a memory shape first, then a tool.
Existing entries verified: 2026-08-30 UTC; new entries verified: 2026-10-15 UTC
Local or self-hosted does not mean zero model or storage cost. When evaluating a memory tool, test update and deletion, not only write and search. Also test user/workspace isolation and whether malicious content can poison future behavior.
| Group | Project / document | Rating | Use | Limits |
|---|---|---|---|---|
| Memory layer | ⭐⭐⭐⭐⭐ | Best for: First cross-session memory What you can learn: library, server, cloud, and search lifecycle | Status/limits: Apache-2.0; distinguish OSS from managed capabilities | |
| ⭐⭐⭐⭐ | Best for: Teams already using LangGraph What you can learn: hot-path/background memory | Status/limits: MIT; understand the LangGraph store first | ||
| ⭐⭐⭐⭐ | Best for: Understanding the Letta product boundary first What you can learn: current installation, docs, and source locations | Status/limits: Landing page; the retired V1 server remains only on the archive branch | ||
| ⭐⭐⭐⭐ | Best for: Building a stateful agent or App Server What you can learn: agent harness, git-backed MemFS, persistent identity | Status/limits: Current source; a product-oriented harness, not a general memory database | ||
| ⭐⭐⭐⭐ | Best for: Using the memory that already exists on disk What you can learn: episodic memory built from coding-agent session files; lexical (BM25) retrieval with no embeddings by default, plus optional semantic recall through deja embed | Status/limits: MIT; local by default. A configured remote embedding endpoint receives the indexed text, with credentials already redacted; a local Ollama or LM Studio keeps it on the machine | ||
| ⭐⭐⭐⭐ | Best for: Readers comparing explicit memory operations What you can learn: retain / recall / reflect: store, retrieve, and reason over memories to answer | Status/limits: MIT; do not treat vendor benchmarks as your own quality evidence | ||
| ⭐⭐⭐ (advanced) | Best for: Teams organizing large agent contexts What you can learn: Filesystem-like organization for resources, memories, and skills | Status/limits: Main project: AGPL-3.0. CLI/examples are separately licensed. The README adds rewriting language. | ||
| ⭐⭐⭐⭐ | Best for: Memory-governance readers What you can learn: Chat Memory, Skill, LLM-Wiki, CodeGraph; owners, versions, visibility, agent bindings | Status/limits: MIT; SQLite by default; extraction uses a model endpoint. Check network and data access | ||
| Time and relationships | ⭐⭐⭐⭐⭐ | Best for: Applications whose relationships change over time What you can learn: bi-temporal facts, temporal graphs | Status/limits: Apache-2.0; requires a graph database and governance | |
| ⭐⭐⭐ | Best for: Teams evaluating Zep Cloud What you can learn: integration and examples entry point | Status/limits: The former Community Edition is legacy/deprecated | ||
| ⭐⭐⭐⭐⭐ | Best for: Readers learning concepts first What you can learn: thread state, stores, three memory types | Status/limits: Framework docs; concepts transfer, APIs are version-specific | ||
| Research and evaluation | ⭐⭐⭐⭐⭐ | Best for: Researching agent-memory architecture What you can learn: working, episodic, semantic, procedural memory | Status/limits: An analytical framework, not an installable product | |
| ⭐⭐⭐⭐⭐ | Best for: Researching reflection and memory retrieval What you can learn: recency, importance, relevance | Status/limits: Classic research, not a production standard answer | ||
| ⭐⭐⭐⭐ | Best for: Readers researching feedback from experience What you can learn: verbal feedback and the next attempt | Status/limits: Reflection becomes cross-session memory only after durable storage | ||
| ⭐⭐⭐⭐ | Best for: Developers testing memory quality What you can learn: datasets and a rerunnable evaluation entry point | Status/limits: Vendor-maintained; add your own isolation/deletion tests |
Self-hosting does not guarantee offline use: extraction/Proxy use configured models; Tencent Cloud is optional. The Docker example disables Gateway Bearer auth; some read-only Knowledge routes need no service key. Avoid untrusted networks; check consent, providers, and retention/deletion before imports. deployment configuration / service authentication boundaries. Docs/config reviewed; performance and isolation untested.
Read OpenViking's LICENSE before integration.
✅ Self-check¶
- I do not treat Chat History, Context, RAG, and Memory as the same thing.
- Every long-term memory has an owner, source, time, and deletion method.
- I can explain the difference between semantic memory and semantic search.
- I have tested updates, deletion, expiry, and cross-user isolation, not only writing and search.
- Sensitive data is not written by default, and users can see and control what is saved.