LoraxBench：面向20种印度尼西亚语言的多任务、多语言基准测试套件
LoraxBench: A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languages
AI总结：
本文提出LoraxBench基准，覆盖20种印度尼西亚语言和6类任务，评估多语言及区域LLM，揭示低资源语言性能差距及语体对模型的影响。
摘要：
作为世界上人口最多的国家之一，印度尼西亚使用着700种语言，但其自然语言处理进展却相对滞后。我们提出了LoraxBench，这是一个聚焦于印度尼西亚低资源语言的基准测试，涵盖6类不同任务：阅读理解、开放域问答、语言推理、因果推理、翻译和文化问答。我们的数据集覆盖20种语言，并为其中3种语言增加了两种语体（formality registers）。我们评估了多种多语言及区域聚焦的大语言模型，发现该基准具有挑战性。我们注意到印度尼西亚语与其他语言（尤其是低资源语言）之间的性能存在明显差距。与通用多语言模型相比，使用区域特定模型并未表现出明显优势。最后，我们表明语体的变化会影响模型性能，尤其是社交媒体中不常见的语体，例如爪哇语的高级敬语“Krama”。
英文摘要：
As one of the world's most populous countries, with 700 languages spoken, Indonesia is behind in terms of NLP progress. We introduce LoraxBench, a benchmark that focuses on low-resource languages of Indonesia and covers 6 diverse tasks: reading comprehension, open-domain QA, language inference, causal reasoning, translation, and cultural QA. Our dataset covers 20 languages, with the addition of two formality registers for three languages. We evaluate a diverse set of multilingual and region-focused LLMs and found that this benchmark is challenging. We note a visible discrepancy between performance in Indonesian and other languages, especially the low-resource ones. There is no clear lead when using a region-specific model as opposed to the general multilingual model. Lastly, we show that a change in register affects model performance, especially with registers not commonly found in social media, such as high-level politeness `Krama' Javanese.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
