How LLMs Handle Infinite Context With Finite Memory

Standard Transformer models face a significant memory challenge with long context windows because the KV cache grows linearly with the input sequence. A method called Infini-attention addresses this by combining standard local attention with a compressive global memory that stores a fixed-size summary of past history. The technique processes input in segments, using local attention for high detail and updating the global memory with compressed information to avoid redundancy. When generating output, the model dynamically mixes the detailed local context with the summarized global history, enabling it to handle virtually infinite context with a fraction of the memory.