Why do capable AI agents still fail to deliver finished work — and how do I measure the gap?

October 19, 2026 · 5 min read

SentX SI

Because the tests they pass measure task-solving, while delivery demands something else: finding out what the customer actually needs, deciding under incomplete information, and shipping a system that holds up in a live environment. The distance between the two is no longer anecdotal. FDE-Gym, a benchmark released on arXiv in mid-October 2026 under the title "The Last Mile of AI Delivery," puts a number on it. Its best-performing configuration — the top of 13 model configurations drawn from seven providers — averaged 0.56 on a scale where 1 is the reference solution and 0 is leaving the system untouched, and it fully resolved only 14 of the 95 delivery tasks.

Source: FDE-Gym: The Last Mile of AI Delivery (arXiv:2610.22350)

The second number is what makes the result hard to dismiss. On the 23 tasks where that same best configuration managed to extract every key fact from the stakeholders, it still reached the 0.8 uplift bar on only 10. Knowing everything that mattered did not produce delivery. The failure splits into two stages: discovering unstated constraints, which the paper identifies as the primary bottleneck, and converting known facts into a working system, which fails on its own.

Source: FDE-Gym: The Last Mile of AI Delivery (arXiv:2610.22350)

How the benchmark works

FDE-Gym builds 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real delivery projects. The scenarios read like a job board: a hospital triage chat, a fund-operations agent clearing reconciliation breaks, EV-charging driver support pinned to a fixed weak model, an AI night responder for a B2B invoicing API. Agents enter a simulated customer environment with scripted stakeholders and a modifiable, production-like system, then have to clarify ambiguous requirements, uncover unstated constraints, and deliver a working solution.

Source: FDE-Gym: The Last Mile of AI Delivery (arXiv:2610.22350)

Grading mirrors how a customer judges. Each delivery is scored by how much it improves the customer's own success metric, normalized between 0 (the system unchanged) and 1 (the reference solution), with penalties for stakeholder interactions — messages sent, trial runs consumed, waiting time incurred. A task counts as resolved only at a score of at least 0.8 with no incident. An online platform lets humans practice the same skills under the same grading criteria, so the bar is not set exclusively for machines.

Source: FDE-Gym: The Last Mile of AI Delivery (arXiv:2610.22350)

Measuring the gap in your own pipeline

The grading design doubles as a self-test. Before you commit to shipping an agent-built solution, run these four moves against your own pilot.

1. Write down the customer-side success metric and the acceptance bar before the run. Name the metric the customer cares about, record the baseline (the system as found is your zero), and fix in advance what counts as delivered — FDE-Gym's bar is 0.8 of the reference uplift with no incident. If "done" is defined after the fact, the pilot cannot measure itself.Source: FDE-Gym: The Last Mile of AI Delivery (arXiv:2610.22350)
2. Score delivery completion separately from task accuracy. Averages flatter. A mean of 0.56 sounds respectable until you count how many tasks actually cleared the bar: 14 of 95. Track both the average uplift and the resolved count, and treat any task below the bar as unfinished, whatever partial value it produced.Source: FDE-Gym: The Last Mile of AI Delivery (arXiv:2610.22350)
3. Price in stakeholder-contact cost. Every question the agent asks, every trial run it consumes, every hour of waiting deducts from the score. During your pilot, log contacts the way the benchmark logs them and hold them against a budget you agree with the customer up front.
4. Report the gap as a number before committing to ship. The stakeholder conversation changes when you can say "we are at 0.4 against a 0.8 bar, and the missing half splits between constraints we had not found and implementation we had not finished" instead of "it mostly works."

These four moves are an editorial transfer of the benchmark's scoring rules, not a validated method — but they are the same arithmetic the benchmark applies, and they turn a gut feeling about the last mile into a figure you can track.

A runnable reference

You do not need the full 95-task suite to start. Twenty of the original scenarios are public on Hugging Face under DeepLethe/FDE-Gym, complete with graders, reference facts, and task instructions at four brief levels — L0 states the most, L3 the least — so you can dial the ambiguity yourself. The harness lives on GitHub at deeplethe/fde-gym; it needs Python 3.9 or later and nothing outside the standard library, and a case runs in three commands: create the engagement, serve it, grade the delivery. The remaining 75 variant tasks are held out for leaderboard scoring, and the reference solutions are not published. Access comes with terms: the files stay inside your organization, graders and solutions are not posted publicly, and the data stays out of model training corpora.

Source: DeepLethe/FDE-Gym dataset card

Two honest limits frame what these numbers establish. The results are author-reported across 13 configurations from seven providers, and the stakeholders are scripted — disciplined simulations, not real customers with real moods. The paper is a recent preprint, so the figures will move as more configurations are tested. None of that weakens the structural point: the distance between solving a well-specified task and delivering for a customer is large enough to measure, and it survives being handed the answer sheet.

Source: FDE-Gym: The Last Mile of AI Delivery (arXiv:2610.22350); DeepLethe/FDE-Gym dataset card

That last clause is the habit worth importing. SentX's model card states the same discipline for its own product — answers can contain incorrect facts, citations or calculations, so open the cited source and check the supporting passage — and extends it to long-running work: keep a portable checkpoint of checked findings and open questions as a project spans conversations, with the card's own note that the checkpoint builder formats notes without verifying them. Delivery projects run for weeks. Whatever tool you build the checkpoint with, the measurement principle is identical: a number you can check beats an impression you can defend.

Source: Victoria AI Model Card