SEA-HELM：东南亚语言模型的整体评估
SEA-HELM: Southeast Asian Holistic Evaluation of Language Models
AI总结：
本文提出SEA-HELM，一个覆盖菲律宾语、印尼语、泰米尔语、泰语和越南语的东南亚多语言多文化LLM评估套件，包含NLP经典、LLM特定、语言学、文化和安全五大支柱，并提供公开排行榜与评估代码。
摘要：
随着大型语言模型（LLMs）中新能力的迅速涌现，对严格且整合的多语言与多文化基准的需求变得更加突出。尽管现有的LLM基准能够评估LLM在英语以及各种中低资源语言（包括东南亚（SEA）地区的语言）中的特定能力，但迄今为止尚未开发出一个全面且具有文化代表性的东南亚语言评估套件。在此，我们提出SEA-HELM，一个强调东南亚语言的整体语言与文化LLM评估套件，包含五个核心支柱：（1）NLP经典任务，（2）LLM特定能力，（3）东南亚语言学，（4）东南亚文化，（5）安全性。SEA-HELM目前支持菲律宾语、印度尼西亚语、泰米尔语、泰语和越南语。我们还引入了SEA-HELM排行榜，使用户能够以系统且用户友好的方式了解模型的多语言和多文化表现。我们公开了SEA-HELM评估代码。
英文摘要：
With the rapid emergence of novel capabilities in Large Language Models (LLMs), the need for rigorous multilingual and multicultural benchmarks that are integrated has become more pronounced. Though existing LLM benchmarks are capable of evaluating specific capabilities of LLMs in English as well as in various mid- to low-resource languages, including those in the Southeast Asian (SEA) region, a comprehensive and culturally representative evaluation suite for the SEA languages has not been developed thus far. Here, we present SEA-HELM, a holistic linguistic and cultural LLM evaluation suite that emphasises SEA languages, comprising five core pillars: (1) NLP Classics, (2) LLM-specifics, (3) SEA Linguistics, (4) SEA Culture, (5) Safety. SEA-HELM currently supports Filipino, Indonesian, Tamil, Thai, and Vietnamese. We also introduce the SEA-HELM leaderboard, which allows users to understand models' multilingual and multicultural performance in a systematic and user-friendly manner. We make the SEA-HELM evaluation code publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
