Mechanistic View of Transformers: Patterns, Messages, Residual Stream… and LSTMs

A mechanistic view of Transformers reinterprets the attention mechanism by separating it into two conceptual processes: patterns and messages. The pattern (QKT) determines which tokens are relevant, while the message (VO) determines what content to transmit. This perspective also reframes residual connections as a persistent "residual stream" where components additively update the embeddings rather than transforming them. This state-updating mechanism creates a strong analogy to LSTMs, where the residual stream is similar to the cell state and the pattern/message components function like the input and output gates.