Spatial Reasoning in AI Models, Taken Apart

AI models are tested on their spatial reasoning abilities using grid puzzles from the SPaRC benchmark, where even the best models fall significantly short of human performance on harder levels. The analysis deconstructs this failure by evaluating specific skills like perception, path tracking, and rule-following. Findings show that visual perception is a major hurdle, as a model's accuracy can change dramatically based on how a puzzle is drawn, such as with added coordinate labels. Models also frequently fail at the basic task of generating a valid, connected line from start to finish, making illegal moves or crossing their own paths before even attempting to follow the puzzle's complex rules.