Your LLM Has a Curved Space of Paragraphs

Standard transformer positional encodings treat text as a flat sequence, ignoring hierarchical structures like paragraphs. This research proposes a hierarchical rotary positional encoding (hRoPE) that gives paragraphs, sentences, and tokens their own coordinates. Through experiments involving interventions like merging and splitting paragraphs, the model with hRoPE learns to suppress attention across paragraph boundaries. This suppression effect is causally linked to the genuine paragraph structure, and its depth varies across different text corpora, suggesting the model develops a structural understanding beyond simple token distance.