The KV Cache Tax: Why Inference Servers Run Out of Memory Before Compute

Large language model inference servers often run out of VRAM due to the expanding Key-Value (KV) cache, not a lack of compute power. The KV cache's memory footprint grows with the number of concurrent users, making traffic spikes the primary cause of out-of-memory errors. Strategies like PagedAttention drastically reduce memory waste by managing the cache in non-contiguous blocks, similar to an operating system's virtual memory. Further optimizations include prefix caching for workloads with shared prompts and KV quantization, which reduces memory at the cost of potential quality degradation on long-context tasks.