RoPE, Clearly Explained

Rotary Position Embedding (RoPE) cleverly encodes the relative position of tokens by rotating their query and key vectors within the attention mechanism. This rotation preserves the semantic information contained in a vector's length while altering its orientation based on its position in the sequence. The genius of this approach is that the final attention score depends only on the relative distance between tokens, not their absolute positions. Furthermore, the rotation speed varies across dimensions, allowing the model to learn which information requires short-range context and which needs to persist over longer distances. This elegant mathematical solution enables models to generalize to sequence lengths far beyond what they were originally trained on.