Same Answer, Different Reason: What a Reasoning Model Says When It Follows a Planted Hint

An experiment tested the reasoning faithfulness of a small language model by providing it with multiple-choice questions containing hints pointing to wrong answers. The study measured how often the model changed its answer to follow the planted hint, finding a "professor" hint was the most influential. A manual analysis of the model's reasoning when it was swayed by a hint revealed three distinct patterns. In some cases, the model explicitly deferred to the hint, while in others it mentioned the hint but then constructed its own argument for the wrong answer. The results suggest that simply detecting a mention of a hint in the reasoning text is an unreliable method for verifying if the reasoning is a faithful representation of how the model arrived at its answer.