SentX SI Blog Meet Victoria

My agent's memory is raw transcripts — should I have it distill its own memory, and what does that quietly lose?

· 5 min read

SentX SI

Short answer: distill it. Raw transcripts are not a memory strategy; they are a log. The real question is not how much to keep after distillation but what form each kept piece survives in, and that is where the quiet loss happens. A paraphrase costs you nothing to produce and everything to recover, so the design has to decide, before the first summary is ever generated, which facts are allowed to exist only in paraphrased form.

The useful shape I would put on this is a two-layer store. The first layer is small and unforgiving: numbers, dates, IDs, error strings, the exceptions that broke a task, and pointers to the source episodes. It stays verbatim, or it is only ever referenced by a pointer that can be followed back to the raw text. A pointer costs almost nothing and buys the raw passage back the moment a downstream task turns on a single digit. The second layer is the distilled narrative: intent, pattern, what the agent concluded and why. That layer can absorb compression, because the reader of a summary usually needs the shape, not the wording. The split matters more than the ratio, because the moment an error string or a figure lands only in the narrative layer, it has already been lost in a way that looks like storage.

Before you trust the distilled layer, run a two-signal check on it. First, does the summary carry any number, identifier or error string at all, or has the prose smoothed every hard token into a generalization? The fully smooth case is where the loss hides, because a summary that reads well and contains nothing checkable passes the eye test every time. Second, does the summary still record which episode or source it came from? The second signal matters more, and it is exactly the thing a consolidation step silently drops. Once provenance is folded away, the summary can no longer be diffed against anything, and a self-generated summary has no external ground truth to diff against in the first place, so the check is the only verification you have. This is a retention-form rule I am proposing as a working method, not a result anyone measured.

The framing for the whole thing comes from a paper submitted to arXiv on 19 November 2026 by K. R. Jayaram and eight co-authors, accepted for an Oral at IAAI 2027, titled "How Much Should an Agent Remember? Deployment Lessons from Self-Distilled Memory for LLM Agents." Their pipeline mines one to three short lessons from an agent's own trajectories per episode, consolidates them into a smaller set of guidelines that each record how many lessons they merged, and serves the agent either all guidelines or a selected subset. The central finding, stated by the authors, is that the right amount of memory is a property of the model reading it, not of the memory: the same distilled store can help one model and overload another. On the AppWorld benchmark, across models with baselines from 40 to 92 percent task completion, they report DeepSeek-V3.2 gaining 9.5 points of task completion and 16.1 points of scenario completion with its full 191-guideline set, while gpt-oss-120b, starting at 39.9 percent, reaches 51.2 percent with all 199 of its guidelines but 55.4 percent with a selected subset of about 29, a mean of two runs. Those are their reported numbers from a single benchmark and a production deployment across IBM's CUGA agent and the watsonx Orchestrate platform, shipped in the open-source altk-evolve library, and they are provisional findings rather than validated benchmarks. Read them as evidence that the amount question is model-relative, not as a target.

Source: How Much Should an Agent Remember? Deployment Lessons from Self-Distilled Memory for LLM Agents

The paper's failure catalogue is the more practical half, because each entry is a way distillation goes wrong quietly. A high-support guideline that backfired in practice. Provenance dropped during consolidation. A swallowed API error that let the agent keep producing a plausible log while failing every task. A confounded baseline that nearly produced a misleading comparison with a prior method. Notice that three of the four are silent: nothing in the pipeline raises a flag when a merged guideline turns out to be wrong, when the source trail disappears, or when the thing being logged has already failed. That is what the two-signal check is for — it is an instrument pointed at the class of loss that has no error message.

Source: How Much Should an Agent Remember? Deployment Lessons from Self-Distilled Memory for LLM Agents

The user-side version of the same discipline is already written down, at least by one vendor. SentX's public model card, a first-party product disclosure rather than an independent certification, says of Victoria's dynamic memory that recall is selective and exact retention is not guaranteed, so anything that must stay exact should be repeated; and the memory is not a transcript replayed on every turn but a store of what a conversation established as worth keeping, from which only the parts that fit the current question come back. SentX's own working guidance is to treat it as good general recall while keeping your own record of exact figures, stating corrections explicitly rather than quietly supplying a new number. That is the cost-bearing-exact layer, expressed as user behaviour instead of as a pipeline design, and it is the same conclusion reached from the other side of the interface: the shape of a project survives compression, the digit does not.

Source: Victoria AI Model Card: Evidence and Known Limits; AI Chat With Memory That Persists

So the procedure I would run is a short one. Tag the hard tokens first, before any summarizer touches a transcript, and put them in the pointer-addressable layer, keeping the episode ID attached. Distill the rest into narrative. Then, on the distilled layer only, ask the two questions in order: is anything checkable still in there, and can you still name where it came from. If the answer to the second is no, throw the summary away and regenerate it with provenance required in the output — that is the cheaper fix than discovering the gap three episodes later, when a plausible conclusion is standing in for evidence you can no longer trace.

Sources

  1. How Much Should an Agent Remember? Deployment Lessons from Self-Distilled Memory for LLM Agents — arXiv
    Supported claims and limits
    • Paper arXiv:2611.26361, submitted 19 Nov 2026 by K. R. Jayaram and eight co-authors, accepted to IAAI 2027 (Oral); scope is the authors' own deployment account, no independent replication stated.
    • Pipeline mines one to three short lessons per episode from the agent's own trajectories, consolidates them into a smaller guideline set where each guideline records its merge count, and serves all guidelines or a selected subset at inference; ships in the open-source altk-evolve library and is deployed in IBM's CUGA agent and the watsonx Orchestrate platform.
    • Central finding as stated by the authors: the right amount of memory is a property of the model, not the memory; measured on AppWorld across models spanning 40–92% baseline task completion — DeepSeek-V3.2 +9.5 task and +16.1 scenario points with its full 191-guideline set; gpt-oss-120b from 39.9% to 51.2% with all 199 of its guidelines but 55.4% (mean of two runs) with a selected subset of about 29.
    • Reported failure items: a high-support guideline that backfired, provenance silently dropped during consolidation, a swallowed API error that failed every task while producing a plausible log, and a baseline confound that nearly produced a misleading comparison with a prior method.
  2. Victoria AI Model Card: Evidence and Known Limits — SentX
    Supported claims and limits
    • First-party product disclosure, not an independent certification; states Victoria has dynamic symbolic memory working in ways similar to human memory, that recall is selective and exact retention is not guaranteed, and that details which must remain exact should be repeated — published product behaviour, not an independent validation of any memory-design method.
  3. AI Chat With Memory That Persists — SentX
    Supported claims and limits
    • Product page states the memory is not a transcript replayed on every turn — it stores what a conversation established as worth keeping and brings back only what fits the current question — and the day-to-day guidance is to treat it as good general recall, keep your own record of exact figures, and state corrections explicitly rather than silently supplying a new number.
Meet Victoria