Fix the Image, Not the Prompt

The performance of vision-language models on visual tasks is often more significantly impacted by changes to the input image than by prompt engineering. Experiments using the SPaRC grid puzzle benchmark demonstrated that accuracy gains from altering how an image is drawn or labeled far exceeded improvements from rewording prompts or adding examples. The most effective image modifications were targeted labels that explicitly identified key components mentioned in the task, such as marking the 'start' and 'exit' or printing coordinates in each cell. This suggests that the primary failure point for vision models can be in perception and linking visual elements to task-specific language, rather than in reasoning. Therefore, a more effective strategy for improving performance is to first diagnose and fix what the model fails to see in the image before attempting to optimize the text prompt.