My long outputs slow down near the end — is that my model, or my serving setup?
5 min read
Both, and the shape of the lateness is what tells you which. A clean separation of the two layers comes from a paper submitted to arXiv on 13 November: VelServe, by Ishaan Datta and colleagues at Sprinklr and Microsoft, which studies why tokens arrive late in batched serving and how to keep them from arriving so. Its diagnosis splits your symptom into two causes, and each has a recognizable signature.
Source: [2611.18392] VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling
The serving-side signature is the occasional big gap. In a batched engine, every decode step runs the whole batch together — one forward pass gives every active request one token. When a step carries more prefill work than it can absorb, a chunk of a newly arriving prompt, or when the host briefly stalls, that step runs past the time-between-tokens target, and every request in the batch receives a late token at once. The authors call this a decode jitter. The arithmetic is what makes long outputs feel slow: jitters are rare per step, common per request. If just one step in a hundred is late, a 300-token answer meets at least one with about 95 percent probability, and real stalls arrive in bursts, so one burst can hand a single response more late gaps than its percentile target allows. Even a constant hazard rate feels like end-of-output slowdown, because the later you are, the more stalls you have already accumulated. In a DeepSeek-V2-Lite run under two SLO-aware schedulers, jitters stayed out of each request's P90 token gap but reached its P99: at a 150 ms target, 89–98 percent of requests met the bound at P90, only 62–74 percent at P99. This is a scheduling artifact of how prefill work is admitted into decode steps, not the model degrading. It also survives the usual fixes: chunked prefill and prefill/decode disaggregation remove long generation stalls but not the jitter, because a token budget caps the work in a step, not the step's duration.
Source: VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling (full text)
The model-side signature is the steady climb. The same paper measures the other effect: under GQA or MLA attention, every token attends to its request's whole history, so a decode deep into a long answer costs more than the same work early on. Your typical per-token time drifts upward across the response, smoothly, on every request. Under compressed attention, as in the DeepSeek-V4 family, decode cost barely grows with context and a decode weighs about as much as one prefill token; on such a model, a gradual late slowdown is not explained by attention and points back at the serving stack.
Source: VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling (full text)
How to tell which one you have
VelServe is a scheduler, so the check below is a reading method I would propose for your own deployment — a suggestion, not a finding of the paper. Log the arrival gap for each token on your streaming path. Split each completed output into thirds by token position, and compare the tail of the gap distribution across thirds, not the mean: the mean is exactly what spreads one long pause over hundreds of short gaps. If the last third has the same tight bulk but more outliers, and those outliers cluster in wall-clock time across concurrent requests, you are seeing jitter — scheduling or host stalls. If the typical gap itself climbs smoothly from first third to last, you are seeing step-time growth — attention cost, and only on GQA or MLA-class attention. A third check is almost free: rerun the same workload with a single request in flight. Cross-request admission jitter disappears at batch size one; genuine step-time growth stays.
What the fix buys, and where it stops
VelServe's answer is to calibrate each step against the measured tail of its own iteration times: the SLO determines how rare a late step may be, and the scheduler leaves that much headroom in every step. The headroom slows prefill, so it serves the shortest waiting prompt first to recover the first-token cost. Under a 150 ms per-request P99 target, it sustains 2.4–2.9 times the SLO goodput of the strongest baseline on Qwen3-8B and DeepSeek-V4-Flash, and at least 4.4 times on DeepSeek-V2-Lite; on a disaggregated eight-GPU deployment it meets the SLO for 35 percent of requests, against 15–16 percent for the baselines. The scope is worth carrying with the numbers: three models covering the three attention families, colocated runs on an H100, disaggregated runs on a node of eight RTX PRO 6000 GPUs, built on vLLM 0.30. The authors' own limits: calibration tracks shifts in the stall distribution with a lag, some of the longest prompts still time out under sustained load, and the disaggregated admission gate was evaluated on one prefill/decode allocation on one node — it helps only when your TBT bound is shorter than a join, which took 170–185 ms in their deployment.
Source: [2611.18392] VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling; VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling (full text)
The habit to keep is the last one. Victoria's model card, which SentX publishes for its own assistant, carries a known-limits line that travels further than the product: open cited sources and check the supporting passage. That is exactly the move a latency benchmark demands. A goodput headline is only as portable as the measured conditions behind it — the SLO target, the workload mix, the hardware, the engine version. Before you swap a scheduler on the strength of a paper or a product page, find the setup section and read what was actually measured. The diagnostic above costs an afternoon of logging; trusting a headline number costs a migration.
Source: Victoria AI Model Card | Capabilities, Evidence and Limits
Sources
- [2611.18392] VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling — arxiv.org
Supported claims and limits
- VelServe paper arXiv:2611.18392, submitted 13 Nov 2026, by Ishaan Datta, Alind Khare, Koushik Kumar Dey, Ratnesh Jamidar, Anand Kumar Singh
- Decode jitters delay every request in the batch; long responses span hundreds of steps, so even rare overruns cause many SLO violations
- VelServe calibrates each decode step's prefill work from the measured tail of its own iteration latency; headroom slows prefill, so it serves the shortest waiting prompt first
- Under a P99 TBT target, VelServe sustains 2.4-2.9x the strongest baseline's SLO goodput on Qwen3-8B and DeepSeek-V4-Flash and at least 4.4x on DeepSeek-V2-Lite; with prefill/decode disaggregated on eight GPUs it meets the SLO for more than twice as many requests as baselines
- VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling (full text) — arxiv.org
Supported claims and limits
- Authors affiliated with Sprinklr (Datta, Dey, Jamidar, Singh) and Microsoft (Khare)
- A decode jitter is a decode step running longer than the TBT target; jitters come from how prefill work is admitted into decode steps, and persist under chunked prefill and disaggregation because a token budget bounds work per step, not step latency
- Illustrative arithmetic (Appendix A): if jitters were independent and hit 1% of steps, ~95% of 300-token answers meet at least one; real stalls arrive in bursts
- Figure 3 runs (DeepSeek-V2-Lite, two SLO-aware schedulers, 150 ms bound): 89-98% of requests meet the bound at P90, only 62-74% at P99
- Under GQA and MLA every token attends to the request's whole history so deep decodes cost more; under compressed attention (DeepSeek-V4) cost barely grows with context and a decode costs about one prefill token
- Evaluation scope: three attention families (Qwen3-8B GQA, DeepSeek-V2-Lite MLA, DeepSeek-V4-Flash compressed), aggregated runs on one H100 per model, disaggregated runs on a node of eight RTX PRO 6000 GPUs, all in the same vLLM 0.30 build; disaggregated attainment 35% vs 15-16%
- Author-stated limits: calibration tracks shifts in the stall distribution with a lag, some longest prompts still time out under sustained load, and the decode gate was evaluated on one P/D allocation on one node; it helps only for TBT bounds shorter than a join, which took 170-185 ms in their deployment
- Victoria AI Model Card | Capabilities, Evidence and Limits — sentx.ai
Supported claims and limits
- First-party model card (reviewed 23 Oct 2026) known-limits section states answers can contain incorrect facts and instructs users to open cited sources and check the supporting passage