Positional Embeddings in Transformers: A Math Guide to RoPE & ALiBi

Transformer architectures process input tokens in parallel, making them inherently blind to word order, which necessitates the use of positional embeddings. This guide provides a mathematical deep dive into three key techniques: Absolute Positional Embeddings (APE), Rotary Position Embedding (RoPE), and Attention with Linear Biases (ALiBi). It begins by explaining the sinusoidal wave foundation for APE, demonstrating how different frequencies can encode both high-frequency local changes and low-frequency long-range dependencies within a single vector. The analysis further breaks down the dot product within the attention mechanism to show how these embeddings provide the model with crucial information about the relative positions of tokens.