Science Explained/Brief
Calibrated router for disaggregated LLM serving shows goodput gains in measured tests
A preprint reports a router that estimates completion time per instance using prompt length, predicted output length, KV cache pressure, and SLO class. Tested on eight NVIDIA A40 GPUs, it achieved the highest mean goodput across three bursty traces.
BriefPublished 16 September 20261 min read1 linked source · 7 checked facts
The study compares a calibrated router against round robin, least loaded, and a length heuristic. On eight NVIDIA A40 GPUs running vLLM engines, it achieved mean goodput of 0.864 versus 0.835 to 0.847 for the baselines. It beat round robin and the length heuristic on all three traces, and least loaded on two; on the third it trailed by 0.003, within run to run noise. Hardware calibration was crucial: simulator derived constants cost 4.5 goodput points and about 40 percent of the tail latency advantage. Benefits grew with decode pool size and traffic heterogeneity but vanished with three instances. Under extreme scarcity, greedy cost minimization concentrated requests, while blind spreading performed better. With calibrated costs, the router matched round robin goodput using six GPUs instead of seven.
Our view
The measured gains are modest and depend on hardware calibration, so the approach's practical advantage may be limited to larger, more heterogeneous decode pools.
What the reporting says: Across three mixed, bursty arrival traces, the calibrated router achieves the highest mean goodput at 0.864, compared with 0.835 to 0.847 for round robin, least loaded, and a length heuristic, and shows the lowest variance across traces.
