How agent memories work

How agent memories work

When people say an AI agent "has a memory," they usually mean one of two very different things: the model's context window (what it can see right now) or a persistent store (what it can look up later). Understanding the difference is the key to understanding what agents can and cannot remember — and why "memory" is the most over-claimed feature in the agent ecosystem.

The context window is not memory

A language model's context window is the set of tokens it can attend to when generating a response. It's like a desk: everything on it is visible right now, and everything not on it is invisible. The desk is large — modern models have very large context windows — but it is finite, and it resets between sessions unless you explicitly carry content forward.

The practical consequence: an agent that doesn't persist anything "forgets" everything between conversations, even though its underlying model is identical. Persistence is an engineering decision, not a model property.

What persistent agent memory usually is

Most agent frameworks implement memory as a small set of storage patterns:

1. Facts and profile entries. Simple key-value or structured records — "user's name is Alex," "prefers concise replies," "project codename is Peregrine." These are cheap, exact, and the most reliable form of memory. When an agent "remembers" a fact you told it, this is usually what happened.

2. Conversation history. Past exchanges saved and re-injected into the context window (fully, or summarized) when a new session starts. This is what gives the illusion of continuity between chats. Summaries trade fidelity for space — a long history that gets compressed will lose details.

3. Vector / semantic memory. The interesting one. Text (messages, notes, documents) is converted into embeddings — high-dimensional numerical vectors that capture meaning — and stored in a vector index. When the agent needs to remember something, it converts the current question into an embedding and retrieves the most similar stored passages. This is _retrieval_, not recall: the agent doesn't "know" the fact, it finds the passage most likely to contain it. The ReAct paper (arXiv:2210.03629) describes the reasoning-and-acting loop that makes this retrieval-driven style of agent work, and OpenAI's practical guide to building agents (PDF) walks through the retrieval patterns in plain language.

4. Files and notes. Some agents maintain a workspace of documents the user (or the agent itself) writes — notes, task lists, saved research. Memory, in this sense, is just a folder with good indexing.

Real frameworks combine these. A typical agent will keep a profile file, a conversation log, and a vector index over notes — each serving a different recall need.

What memory is NOT

Three honest corrections, because agent marketing loves this topic:

• Memory is not perfect recall. Retrieval can miss the relevant passage; summaries lose details; models can misapply what they retrieved. An agent that "remembers" something can still be wrong, and it can be confidently wrong.
• Memory is not understanding. Storing a fact and _using_ it correctly are different steps, and the second one fails independently of the first.
• Memory is not security. Whatever the agent persists is stored somewhere. If that store is not encrypted and isolated, "the agent remembers my passwords" is a liability, not a feature. Never store secrets in agent memory; store them in the platform's secret handling.

Privacy and isolation

Because memory is stored state, isolation is the defining question for hosted agents: whose agent can read what? On HostAgentics, every agent runtime has its own isolated memory store — there is no shared agent memory, no cross-tenant access, and the platform itself does not read agent memory. Memory lives in the runtime's own persistent volume, which is included in daily provider snapshots (7-day retention, 14 days with the Resource Boost add-on), so "the agent forgot" is a software problem you can actually recover from. See our subprocessors disclosure for who processes what.

Managing memory like an operator

Practical advice, regardless of platform:

• Prefer facts over prose. Structured profile entries retrieve more reliably than free-text notes.
• Curate periodically. Agent memory accumulates junk; periodic cleanup measurably improves retrieval. Many frameworks expose a way to review what's stored.
• Never rely on memory for critical state. If a fact being forgotten would cause harm, put it in a workflow or a database the agent queries — not in the agent's memory.
• Back up the memory store. Agent memory is the most valuable data an agent holds, and the least likely to be backed up by people who self-host. Treat it like a database, because it is one.

The bottom line

Agent memory is a set of engineering patterns — context management, facts, history, vector retrieval — and each pattern has known failure modes. Understanding which pattern an agent uses tells you what it will reliably remember (facts), what it will approximately remember (conversations), and what it will sometimes miss entirely (anything that depends on retrieval quality). Hosting isolates and backs up the store; the retrieval quality is the framework's job and your curation's job. Both matter, and neither is magic.