Alyah ⭐️: Toward Robust Evaluation of Emirati Dialect Capabilities in Arabic LLMs

A new benchmark named Alyah has been introduced to evaluate how well Arabic Large Language Models (LLMs) understand the Emirati dialect. Existing benchmarks primarily focus on Modern Standard Arabic, failing to capture the linguistic and cultural nuances of regional dialects used in daily life. Alyah addresses this gap with a dataset of 1,173 manually curated multiple-choice questions from native Emirati speakers. These questions test a model's grasp of culturally embedded meanings, local expressions, and heritage-related topics, providing a more robust assessment of real-world language capabilities.