Change One Monty Hall Rule and Five LLMs Still Say Switch

An experiment tested five large language models, including Qwen2.5 and Phi-3.5-mini, on the classic Monty Hall problem and several variants with altered rules. While the models correctly answered the standard puzzle, they consistently failed on the variations, recommending to switch even in scenarios where it was the incorrect strategy. The models' performance suggests they rely on memorized patterns associated with the problem's structure rather than performing logical reasoning based on the specific rules provided. This failure to adapt highlights a significant limitation in the reasoning abilities of current LLMs when faced with familiar problems that have modified parameters.