Continuous batching from first principles

Large Language Models generate text one token at a time, a computationally intensive process that involves re-evaluating the entire sequence for each new word. At the heart of this capability is the attention mechanism, which allows different tokens within a prompt to interact and influence one another. These interactions are controlled by a crucial attention mask, which dictates which tokens can "see" others, often in a causal way where a word only attends to those that came before it. Understanding this foundation is key to grasping optimizations like continuous batching, a technique designed to maximize performance by processing multiple user requests in parallel.