How to Benchmark LLMs – ARC AGI 3

A new benchmark called ARC AGI 3 tests artificial intelligence with interactive puzzle games that have no instructions, forcing models to learn the rules through pure experimentation. While humans can solve these puzzles with relative ease, even the most advanced frontier LLMs currently score a striking 0%, highlighting a major gap in their reasoning and problem-solving abilities. This failure is likely due to models struggling with long action sequences and a lack of training data for agent-like behaviors where they must learn from trial and error. As future models improve on this benchmark, it will be crucial to distinguish genuine intelligence gains from "benchmark chasing," where models are simply overfitted to pass the test.