Agent Memory Is Not a Database: Start with an Event Log

I am still trying to decide what “memory” should mean in an agent. I did not begin with a database. I began by comparing how different tools use the word.

OpenAI Agents SDK calls a single conversation’s history session memory; LangGraph separates thread-scoped short-term state from long-term data that spans threads. In the material I am comparing, the same word can refer to a complete conversation, a checkpoint, user preferences, a search index, or even a set of reusable Skills.

That boundary is still blurry for me. I keep seeing the temptation to point a vector database at several different problems and expect “memory” to appear on its own. I do not know yet which parts need separate systems.

This article records an experiment that is still in progress. It is not a product benchmark or a blueprint for every production agent. My current working hypothesis is:

I am treating agent memory as a data lifecycle. I want to preserve traceable events first, then use real failures to decide whether summaries, facts, search, or graphs are worth adding.

I am testing when memory becomes context

I am using these as provisional definitions:

Name	What it is responsible for	What it is not
Context	The instructions, messages, tool results, and retrieved data the model actually sees for one inference	All history stored in the backend
Session state	Messages, plans, checkpoints, and pending state for one thread or task	User memory shared across sessions
Agent memory	Data governed by policies for writing, storing, retrieving, assembling, and forgetting, for future use	A particular database product
External knowledge base	Domain knowledge such as documents, policies, code, or product data	Personal records produced through user-agent interactions

Under these definitions, memory and context sit at different layers. Data can remain in the backend for a long time, but it becomes context only when the system selects it, checks authorization, and places it in the current input. Clearing context does not mean the backend data has been deleted.

I am also keeping content types and retrieval methods separate for now. Episodic memory records what happened and when. Semantic memory stores relatively stable facts. Vector search is only one way to find data. An episodic event can live in an event table, full-text index, and vector index at the same time. I am treating these as compatible choices, but I have not tested which combination will be useful.

The five responsibilities I am using as a test map

Once I separate those terms, I can draw the data flow I want to test. Each step asks a different question. Corrections and deletions also need a path back to storage:

This is the map I am testing. I am still checking whether these boundaries are useful in practice.

The first implementation I am testing: bounded context and a durable event log

For a new agent, the first version I am trying has two pieces: a bounded current context and a durable event log.

In this version, the event log stores user messages, model outputs sent externally, tool calls and results, approvals, errors, and references to artifacts. I am not storing the model’s hidden reasoning. Each run gets only the current request, task state, recent events, and data explicitly needed for that request.

request
   |
   +-- append event ----------------------+
   |                                      |
   +-- read task state                    | durable event store
   +-- read recent events within budget   |
   +-- assemble context -> model -> tools-+
                                          |
                                          +-- optional projections later
                                              summary / facts / search / graph

At the moment, I am treating raw events as the canonical record. Summaries, profiles, full-text indexes, embeddings, and graphs are rebuildable projections. The reason for this choice is traceability: an incorrect summary can be rebuilt, and the search method can change without making the first derived form the only truth. I still need to test whether the extra storage and write-back work is worth it.

This choice has limits. It assumes a conventional database can still handle the raw event volume and that the product values auditability, correction, and deletion. For a one-off task with no cross-turn value, I may not need long-term memory at all.

I am testing long context as a baseline

A complete conversation history is the short-term baseline I am trying first. OpenAI Agents SDK sessions retrieve history before a run and append new items afterward. They also let each run limit how many items it retrieves. The backend can keep the full history while the model input stays bounded.

Sending all history to the model costs more as the content grows, and fitting everything into the context window does not mean the model will use it reliably. Lost in the Middle found that the position of relevant information in a long input affects performance, with particular degradation when the information appears in the middle.

In this experiment, sliding windows, compaction, and summaries are possible ways to control more than a hard context limit. They also change the cost, latency, and distraction of each inference. I am treating a summary as a navigation aid, not raw evidence. When an answer concerns amounts, permissions, commitments, or citations, I want the system to return to raw events or an authoritative source.

I am adding a layer only when a failure gives me a reason

I do not know yet whether a profile, vector index, or knowledge graph will be useful in the first version. For now, I am using an observed problem as the reason to try a layer:

Candidate layer	When it is worth adding	Constraint to keep
Summaries	Long threads exceed token or latency budgets, and truncation begins to hurt task success	Link every summary to event IDs; never replace the raw record
Structured facts / profile	The same stable facts are asked for repeatedly, event scanning is unreliable, and a small schema covers most cases	Keep sources, validity periods, and superseded versions; do not let model inference silently overwrite facts
Full-text search	Names, codes, error messages, and exact wording must be found reliably	Synonym paraphrases may not have enough recall
Vector / hybrid search	Volume or natural-language paraphrases mean key lookup and full-text search miss recall or latency targets	Similarity is not relevance; exact IDs still deserve structured queries
Knowledge graph	Real queries repeatedly need multi-hop relationships or corpus-wide aggregation, and evaluation shows the gains offset update costs	Entity resolution, stale relationships, and extraction errors all need governance

Provider-managed conversation state is another option I am comparing. It can reduce the plumbing around sessions and compaction, but it cannot decide fact conflicts, cross-tenant permissions, or deletion of projections for an application. For cases where I need auditability, portability across models, or explicit data governance, I am keeping the canonical event log in the application.

I am testing corrections, deletion, and permissions at write time

One assumption I am testing is that scope belongs in the storage and authorization key, not in a filter applied after vector search. Data across users or tenants should not enter the candidate set first and then depend on a reranker to filter it away.

In the current version, each event retains its source and time. observed_at means when something is said to have happened; recorded_at means when the system received it. Keeping them separate distinguishes what was known at the time from what is believed now. When a correction arrives, I am trying a new version that invalidates the old one instead of directly overwriting the final string.

I am treating retrieved webpages, email, and old memories as data. Storing them does not turn them into instructions. Otherwise, one indirect prompt injection can become persistent memory poisoning across turns. Tool authorization, data isolation, and approval still need to be enforced outside the model. OpenAI’s agent safety guide also recommends isolating untrusted data and retaining tool approval.

Deletion is another part of the test. If data is subject to the GDPR, Articles 5 and 17 set requirements for data minimisation, retention periods, and erasure in specific circumstances. The engineering problem is bigger than deleting one event-table row. Deletion must propagate to summaries, facts, full-text and vector indexes, graphs, caches, and provider state. An immutable event log cannot justify retaining personal data forever.

What I need to measure at each point

The final answer alone cannot tell me whether a problem began at writing, retrieval, or context assembly, so I am separating the checks in the experiment. First I ask whether the information that should be retained was stored, and whether casual or sensitive data was written by mistake. Then I check whether the correct event entered the candidate set and how much stale or irrelevant content came with it. After retrieval, I check whether the token budget or deduplication dropped any of the data that was found. I also want to know whether adding memory improved task success and factual accuracy, and what another layer costs in p95 latency, input-token use, storage, and model spend.

The test data also needs a time split. For each case, I need to set a cutoff, build the store and indexes using only events before it, then test with later queries. Otherwise, an updated summary or search index may see future data. LongMemEval turns long-term interaction memory into evaluable tasks and also treats memory recall and the final answer as separate observations.

The comparison I am using has three baselines: no long-term memory, bounded history only, and the candidate memory architecture. If a new layer does not yield a measurable improvement on the same temporal holdout, I should remove it.

The questions I am using for now

Before I add a memory layer, I start with the use case. What is actually failing because the agent “forgets”? I then check whether the canonical source is chat, a tool system, documents, or model inference, and whether the scope is a thread, user, agent, or tenant. I need to know who can read and write that data, how corrections, expiry, and deletion propagate to every projection, and which temporal-holdout metric must pass before the extra latency and maintenance cost is worthwhile.

These questions will not give an agent “infinite memory,” and I do not expect them to settle the design by themselves. They give me a way to record the source, purpose, and lifespan of each piece of data while I test whether another layer is justified.

For now, my working setup is bounded context and a durable event log. Summaries, facts, vector search, and graphs can come later if the tests show that I need them. I do not know yet whether this is the best option. I am still working through the alternatives by trial and error, checking which ones hold up when the agent fails.