What We Learned by Reproducing 2,200 papers from ICML

A large-scale hackathon was organized to reproduce the claims of over 2,200 papers from the ICML 2026 conference using AI coding agents. This community effort found that 51% of the examined papers had at least one claim verified, while 23% had a claim falsified or contested. The project demonstrates how AI agents can help audit the rapidly growing volume of scientific research that exceeds human review capacity. The results also highlight the adversarial and complex nature of reproducibility, with independent teams sometimes reaching opposite conclusions on the same claims.