KV Cache Is Eating Your VRAM. Here’s How Google Fixed It With TurboQuant.

The KV cache in Transformer models improves inference latency but consumes significant VRAM, creating a bottleneck, especially for large models with long contexts. Google's TurboQuant technique addresses this by compressing the Key and Value (KV) matrices by over 4.5x with near-zero accuracy loss. The method uses a two-stage process involving PolarQuant for compression and Residual Correction to recover lost information. PolarQuant first applies a randomized rotation to the vectors to smooth out outlier values, then uses Lloyd-Max quantization with a pre-computed optimal codebook to efficiently compress the data.