My self-evolving agent produced new tools — how do I know it gained a capability, not just an artifact?
4 min read
The honest answer is that the evidence you are probably looking at does not answer it. A paper posted to arXiv on November 3, 2026 — Artifact-to-Capability: Diagnosing Breaks in Capability Formation of Self-Evolving Agents in a Controlled Environment (arXiv:2611.04952), by Ziqian Ban and eight coauthors — demonstrates the gap directly. The authors evolved three self-evolving agent frameworks from the existing literature, Reflexion, Voyager, and DGM, in the same controlled environment, freestyle Gomoku, with the same base model. All three showed the standard signals the field treats as evidence of progress: artifacts accumulate, behavior changes once the artifacts are loaded, and metrics rise. Yet each showed a distinct break in capability formation.
The paper calls that usual approach signal-based evaluation, and its starting observation is that what these signals are meant to indicate has never been defined. To fill the gap, the authors define capability formation from artifacts as five sequentially dependent properties, two of which form a checkpoint, and introduce three-scale diagnosis that judges those properties at the state, game, and lineage levels. The abstract gives the count and the structure; the individual definitions are in the full paper. What matters for your situation is what the definition makes visible: the three breaks.
Three ways an agent can look better without being better
Reflexion's break is a legalization gain: its pass rate rose only because more of its moves were legal. The agent got better at not making errors, the metric climbed, and playing strength did not form. Voyager's break is a compositional override: its components were sound on their own, but combined, they missed all 20 available one-move wins. Each skill checked out in isolation; the composition failed. DGM's break is loss-only drift: after passing the checkpoint, later versions only lost passing states, while all three signals were still present.
The value of this trio is that it maps to failure modes you can actually meet when your agent writes its own tools. A metric that climbs because a trivial subtask got easier. Individual skills that pass in isolation but break when the agent has to chain them. A version history that quietly gives back what earlier versions had.
One scope note, because it matters: the result rests on three counterexamples in one environment with one base model. It establishes that signal-based evaluation is insufficient, not a general theorem that it can never work. But for a builder deciding whether to trust a new tool, three concrete ways the dashboard can mislead is enough to change what you check.
A check you can run yourself
Here is the minimal check I would propose, and I want to be clear that this is our editorial suggestion, not a finding of the paper: a before/after re-test on fixed inputs. Before you accept a self-modification, pin down a fixed set of inputs — the tasks, positions, or prompts your agent is judged on — and record its behavior on them. After the change, run the identical inputs again and compare the behavior, not the aggregate score. If you did not record a baseline before the change, run the fixed set now and hold it for the next one; the check has power only when the inputs are identical across the comparison.
What you are looking for is a change in reliable behavior in the right direction: previously failing cases now pass, and previously passing cases still pass. The fixed set should include inputs that force the agent to combine multiple tools or skills, because that is where Voyager's break lived. It should include previously passing cases, because that is where DGM's drift hides. If the only thing that moved is a trivial sub-behavior — more legal moves, more files written, more calls to the new tool — the gain is at the artifact level, and the capability has not formed. If nothing on the fixed set changed, the new tool is inert, whatever the dashboard says.
The paper's sequential structure suggests the ordering: check the earlier properties before the later ones, and treat the checkpoint as a gate, not a finish line. DGM passed the checkpoint and then lost ground. Passing once does not mean the capability is stable.
The same discipline applies to vendor claims
This framing is useful outside your own agent too, and here I will flag a first-party interest: SentX SI's own model card, reviewed October 23, 2026, lists self-learning and self-improvement as current product capabilities of its assistant Victoria, and in the same document notes that recall is selective and exact retention is not guaranteed. The card is a first-party disclosure, not independent certification, and that pairing — a stated capability next to a stated limit — is the artifact-versus-capability discipline the paper argues for, applied to our own claims. When you read any self-improving system's claims, including ours, the question the paper hands you is the right one: what reliable behavior changed, and on what fixed evidence?
Source: Victoria: public model card
Before you celebrate the new tools, run the fixed-input re-test and check which of the three breaks your metrics might be hiding.
Sources
- Artifact-to-Capability: Diagnosing Breaks in Capability Formation of Self-Evolving Agents in a Controlled Environment — arXiv
Supported claims and limits
- arXiv:2611.04952, submitted 3 Nov 2026, by Ziqian Ban and 8 other authors
- Reflexion, Voyager, and DGM evolve in freestyle Gomoku with the same base model and all show the three proxy signals (artifacts accumulate, behavior changes once loaded, metrics rise), yet each exhibits a distinct capability-formation break
- The paper names the standard approach signal-based evaluation and observes that what the signals are meant to indicate has not been defined
- Capability formation from artifacts is defined as five sequentially dependent properties, two forming a checkpoint, with three-scale diagnosis at state, game, and lineage levels; the abstract gives count and structure, not individual property names
- Reflexion: legalization gain (pass rate rises only because more moves are legal); Voyager: compositional override (sound components combine to miss all 20 available one-move wins); DGM: loss-only drift (after passing the checkpoint, later versions only lose passing states)
- The counterexamples show signal-based evaluation cannot determine whether a capability has formed — an insufficiency claim from three counterexamples in one environment, not a general theorem
- Victoria: public model card — SentX SI
Supported claims and limits
- The card, reviewed October 23, 2026, states Victoria self-learns and self-improves through experience and interaction and that these are current product capabilities
- The same card notes recall is selective and exact retention is not guaranteed
- The card is a first-party product disclosure, not an independent certification