Continuous Batching Beats Static and Dynamic by Up to 2.91× in vLLM

An analysis of LLM inference serving shows that the scheduling policy is a major factor in throughput. Continuous batching, which refills GPU slots as individual requests complete, can improve throughput by up to 2.91x over static methods, particularly with variable traffic. This gain comes from avoiding idle GPU slots that occur in static batching while waiting for the longest request in a batch to finish. The underlying principle is that the cost of a decode step is dominated by reading model weights, so keeping the batch full maximizes hardware utilization.