A Three-Phase Factual Recall Circuit in Gemma-2B and Gemma-12B-IT

A mechanistic interpretability study uses activation patching to localize factual recall circuits within the Gemma model family. The research identifies a consistent three-phase circuit for how facts are processed: storage in the early-to-mid layers, routing via distributed attention heads, and readout in the final layers. This pattern was found to replicate at scale, appearing in both Gemma-2B and Gemma-12B-IT, with the residual stream being the dominant causal component for storing facts. The study also notes that tokenizer differences can create challenges for cross-model comparisons.