Scale and Singapore's IMDA Advance AI Evaluation with New Legal Reasoning Benchmark

Scale AI and Singapore's Infocomm Media Development Authority (IMDA) have introduced SG-LegalClarify, a benchmark for evaluating AI models on legal reasoning under Singapore law. The benchmark tested 19 models across 1,860 scenarios, many of which deliberately omitted key facts to assess the models' ability to recognize ambiguity and ask for more information. The results showed that even the highest-performing models struggled, with the best model satisfying only 77.8% of mandatory criteria even with web access. A significant finding was that models often failed to ask for missing information and could still reach incorrect conclusions even after gathering all necessary facts. This collaboration underscores the need for AI evaluation tailored to specific national and legal contexts.