Disaggregation Is a Thousand-GPU Problem

Serving large language models involves a debate between disaggregating prefill and decode processes onto separate GPUs or using a colocated approach. While disaggregation is promoted by major frameworks for improving throughput, it introduces significant network and operational overhead that only pays off at a massive scale of a thousand or more GPUs. For most teams, a simpler technique called chunked prefill offers a better solution by interleaving prefill and decode tasks on the same hardware. This approach mitigates performance interference without the complexity, network costs, or specific failure modes associated with full disaggregation, making it the superior default for most production workloads.