Can I distill my agent's planning into a fixed planner-free workflow — and when would that workflow stop working?

November 12, 2026 · 7 min read

SentX SI

Yes — if the procedure is stable, runs at volume, and your logged runs actually cover the branches that matter. ReTrace, a paper from Columbia University and Amazon submitted to arXiv on 10 November 2026 (arXiv:2611.14492), shows the distillation working and quantifies what it costs: on 79 production procedures, compiled workflows complete 93.8% of runs against 96.8% for the original agent, at 24% lower cost per task. The paper is equally candid about where a compiled workflow stops working, and for a decision, that second list is probably the more useful half.

Source: [2611.14492] ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows

What ReTrace does

The setup is a task family: one standard operating procedure written in natural language, run repeatedly over new inputs. The distillation takes the SOP plus a handful of logged agent runs — traces, where each step records the agent's reasoning, the tool called, its inputs, and the result — and builds a reusable workflow in which code, not an LLM, selects the next step. Construction has three stages. Extraction reads each step and pulls out the condition under which the action applies, the tool, and where each input comes from. Aggregation merges those records across runs into a compact rule document, an intermediate representation, consisting of a completion check and an ordered list of guarded rules, each rule citing the trace steps or SOP statements that support it; branches the agent mentioned but never executed are marked unwitnessed. Synthesis generates the executable code or workflow graph from that document. Only synthesis is platform-specific, so the same rule document can be compiled for different orchestration targets.

Source: [2611.14492] ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows; ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows (full text)

Two details matter for a reader planning their own version. At runtime, if the completion check fails and no rule's condition holds, the workflow stops as incomplete rather than asking a model — a visible, auditable failure mode, but only if you watch for it. And individual tools may still call models; planner-free means no model picks the next step, not that no model runs anywhere.

Source: ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows (full text)

Why the agent's reasoning matters

Tool calls alone rarely say why the agent did what it did; the written reasoning often does. The paper's central ablation removes the reasoning from every trace and keeps the rest of the pipeline: completion on the 79-procedure production set drops 8.8 points, task success on the telecom split of tau^2-bench drops 6.2. On SOP-Bench's short, lookup-like procedures the effect flips — 1.7 points better without reasoning — because there is little for the reasoning to add. The practical takeaway is that your logs need the agent's explanations, not just its call records.

Source: [2611.14492] ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows; ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows (full text)

The numbers

On the production platform (79 procedures, sampled by customer usage): 93.8% completion versus 96.8% for the agent — a paired difference the paper reports as not significant — cost falling from $0.127 to $0.096 per task, and median time from 68s to 37s. On tau^2-bench (telecom domain, solo mode without a simulated user): 75.8% task success against the agent's 80.6%, running in 1.3s instead of 170s, roughly 130 times faster; three rounds of repair on training failures lift the two-trace workflow from 66.7% to 81.2%, close to the agent though the confidence interval is too wide to claim parity. On eight of SOP-Bench's 14 domains (the rest excluded for tools that returned answers directly, missing input columns, or stubbed tools): unrepaired ReTrace at 80.6% trails few-shot code generation at 83.6%, but repair from labeled training failures lifts it to 87.8% at 0.40s per task, against the agent's 85.5% in 46.6s — with the caveat that the repair feedback shows each failed input's label, so that comparison is not like-for-like.

Source: [2611.14492] ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows; ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows (full text)

On payback: from measured savings plus a construction cost, the authors estimate the build pays for itself after about 75 executions. For context on why that matters, seven of the ten most frequently executed automations on their platform between April and June 2026 were fixed workflows, each running 297,000 to 528,000 times in three months, while 91.8% of the 5,115 new automations created in that window were agents that plan at runtime.

Source: ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows (full text)

When it stops working

The paper's own limitations are the honest answer to the second half of your question.

1. Steps that must read free text at runtime. Where a step has to interpret open-ended input — a message, a document, a customer comment — the methods that keep a model in the loop win; on SOP-Bench the agent's lead comes mostly from exactly those steps.Source: ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows (full text)
2. SOP or tool changes. The workflow is a snapshot. A change to the procedure or its tools may require rebuilding, and the evaluation covers new inputs to the same procedures, not transfer to new ones.Source: ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows (full text)
3. Branches your traces never covered. The paper draws traces at random, and the draw matters: on a single SOP-Bench procedure, results vary by as much as 50 points across different trace samples. Happy-path logs produce a happy-path workflow.Source: ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows (full text)
4. Completion flags lie. Platform-reported completion can mark failed runs as complete. Web-tool calls succeeded 86.6% of the time for ReTrace and 27.3% for the agent, and some agent runs with no successful tool call still counted as complete — so even the agent's 96.8% overstates real success, and the same audit gap applies to a compiled workflow.Source: ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows (full text)
5. Recovered conditions may be wrong. The construction checks confirm the code loads and makes no planning calls; they do not verify that the recovered conditions are correct.Source: ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows (full text)
6. More traces are not always better. On tau^2-bench the build rate falls from 100% at k=16 to 31% at k=32 when full traces go to the generator; on the production set, nine full traces lower completion to 81.4%, below zero-shot. The compact rule document is what keeps the generator's input small.Source: ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows (full text)

A pre-flight check for the decision

The paper supplies the ingredients but not a decision rule, so here is one I would apply, built from its own numbers — a proposed editorial method, not something the authors measured or tested.

Before you compile, three gates. First, volume and stability: with payback around 75 executions, this is for high-frequency procedures that have not changed recently; a SOP in flux means constant rebuilds. Second, branch coverage rather than raw trace count: the 50-point variance across samples is the price of random draws, so before building, check which branches of the SOP your logs actually touched and pull traces that cover the ones that matter. Third, an audit path for tool outcomes: if you cannot tell from your own data whether a run that "completed" actually succeeded, you are not ready to trust completion as a metric for either the agent or the compiled workflow.

Source: ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows (full text)

At runtime, watch four signals that the frozen pipeline is hitting its edge before it breaks: workflows stopping as incomplete, since each stop is a report of a branch no rule covered; a rising share of runs where tool outcomes show failure behind a completion flag; any change to the SOP text or tool schemas, which should trigger a rebuild review; and a growing count of unwitnessed rules in the rule document, since that count is the workflow telling you where it is guessing.

Source: ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows (full text)

The pattern behind it

The shape that recurs here is delegation with a checkable handoff: an agent does the work while observed, a distillation step compiles what was observed into something fixed, and a human reviews the compiled artifact before it runs unattended. The paper's own ethics section lands in the same place — owners should review the recovered conditions, restrict tool access, and require human review for consequential actions, because a compiled workflow repeats an incorrect condition across thousands of cases. That review step is the whole safety case, and it is cheap precisely because the rule document is small and cited.

Source: ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows (full text)

The same describe-build-review shape sits behind our own product work. Photon, SentX SI's AI coding agent, has a team of agents plan, build, and deliver what you describe, with the result reviewed before it is accepted; ReTrace is the process-level version of that handoff, where the agent's judgment is frozen into code and the review moves from checking every output to checking the rule document once. Disclosure: Photon is a SentX SI product and this article is published by SentX SI; the ReTrace paper is independent Columbia and Amazon work, and nothing in it is SentX's.

For the decision itself: if your procedure is high-volume, stable, and mostly deterministic lookups and conditionals, and you can audit tool outcomes, compile it — and keep the rule document, since it is the only place the workflow's assumptions are written down. If a step must read and judge free text, leave an LLM in that step and compile the rest around it; the paper's design allows it, and that hybrid is where the agent's 96.8% and the workflow's 1.3 seconds stop competing.

Source: [2611.14492] ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows; ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows (full text)