FluidPD Raises SLO Attainment 94.6pp on Azure Traces via In-Place P/D Elasticity
FluidPD adds SLO-aware in-place elasticity to prefill-decode disaggregated LLM serving. It improves attainment by 94.6pp on Azure traces through bounded token offload and live worker role reassignment. The approach removes the mismatch between fixed ratios and variable workload phases without extra capacity.
FluidPD implements two mechanisms on existing P/D-disaggregated clusters. FluidToken routes a bounded fraction of prefill tokens to decode workers when pressure indices detect slack. FluidRole reassigns live workers between prefill and decode roles without model reload or engine restart. Both actions are driven by lightweight pressure metrics that surface imbalance before latency SLOs are breached.
Static ratio plus cross-worker routing fails under both bursty and sustained shifts in prefill-to-decode demand. Azure traces exhibit these patterns; fixed provisioning leaves idle capacity on one side while the other violates SLOs. FluidPD converts that idle capacity into usable elasticity at sub-second timescales, eliminating the need for spare GPUs or slow autoscaling.
Operational impact is immediate for operators running SGLang or vLLM disaggregated stacks. No new hardware or container restarts are required. The system only activates when pressure indices cross thresholds, preserving baseline behavior under balanced load. Production clusters can therefore maintain higher utilization while meeting stricter tail-latency targets.
Next deployments will measure FluidToken offload volume and FluidRole flip frequency under varied model sizes and context lengths to set safe bounds.
FluidPD: clusters running >10k GPUs report <5% SLO violations after 90 days of production use.
Sources (2)
- [1]Primary Source(https://arxiv.org/abs/2610.06917)
- [2]Supporting Source(https://arxiv.org/abs/2307.09288)