VisualToolBench: Testing the Limits of AI Vision

Scale AI's VisualToolBench (VTB) is a new benchmark designed to test how well models can "think with images" by actively manipulating them to solve problems. The results showed that even top-tier Multimodal Large Language Models performed poorly, with no model achieving over 19% correctness. The primary reason for these failures was not flawed logic but fundamental errors in visual perception, where models could not correctly interpret or extract information from images. The study also found that simply having access to image manipulation tools did not guarantee models could use them effectively or efficiently.