Which tokens does a hybrid model predict better?

Hybrid language models, which combine attention and recurrent layers, are compared against standard transformers to determine which types of tokens each architecture predicts better. The analysis shows that hybrid models have an advantage on meaning-bearing content words like nouns and verbs, which require tracking the evolving context. In contrast, transformers excel at predicting tokens that are verbatim repetitions from earlier in the input, where the attention mechanism's ability to directly copy is a key strength. This research suggests that using filtered losses on specific token types provides a more nuanced evaluation of architectural differences than a single overall loss score.