🇵🇭 FilBench - Can LLMs Understand and Generate Filipino?

FilBench is a new, comprehensive evaluation suite designed to assess the capabilities of Large Language Models (LLMs) for Philippine languages, including Tagalog and Cebuano. It addresses the lack of systematic evaluation for these languages despite high user engagement with AI tools in the region. The benchmark is divided into four main categories: Cultural Knowledge, Classical NLP, Reading Comprehension, and Generation, encompassing 12 distinct tasks. Built on the Lighteval framework, FilBench provides a standardized method for testing models on culturally and linguistically relevant data. Initial results from evaluating over 20 LLMs indicate that while proprietary models like GPT-4 perform best, training region-specific models with local data remains a promising direction.