Why a Tiny GPT Can Know Two Facts and Still Not Connect Them

Language models can suffer from a "Two-Hop Curse," where a model that learns "A leads to B" and "B leads to C" separately fails to infer the connection "A leads to C." An experiment with a tiny GPT model demonstrates this phenomenon by training it on invented facts, resulting in perfect one-hop recall but zero accuracy on two-hop questions. A proposed fix called the "identity bridge," which adds training examples mapping an entity to itself, was tested to see if it could connect the two facts. While this fix initially enabled the model to perform two-hop reasoning, the ability degraded significantly with continued training, suggesting the solution is temporary and sensitive to training parameters.