BenchMIRT: What are LLM benchmarks actually measuring?

BenchMIRT is a new method for auditing LLM benchmarks by analyzing performance on individual questions. It applies multidimensional Item Response Theory (MIRT) to identify the underlying capabilities, such as safety and general reasoning, that each prompt actually measures. The analysis reveals that some benchmarks mix signals, with some safety-oriented benchmarks aligning more with general reasoning than their stated purpose. This approach also helps identify the most informative questions, enabling more efficient evaluations with fewer prompts while preserving accuracy.