Indi-RomCoM：评估LLM在罗马化印度语-英语混合指令上的基准测试
Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions
AI总结：
提出Indi-RomCoM基准，用于评估大语言模型在罗马化印度语-英语混合指令上的表现，涵盖7个任务、4种语言和3种混合强度，发现模型性能随混合密度增加而下降。
摘要：
罗马化代码混合（RCM）是指双语使用者将本地语言与英语流畅地混合使用罗马字母书写，已成为多语言社区中的主要交流形式。虽然大语言模型（LLM）在单语和本地脚本基准测试中表现强劲，但它们遵循指令并推理基于RCM的内容的能力尚未得到充分探索。为此，我们引入了Indi-RomCoM基准，以促进对印度罗马化代码混合指令的系统评估。我们的基准涵盖七个指令遵循任务、四种广泛使用的印度语言和三种受控的代码混合强度级别。我们在零样本和少样本设置下广泛评估了一系列LLM，包括专有模型、开源模型和专注于印度的模型。LLM在RCM指令上始终表现不佳，且性能随代码混合密度的增加而下降。此外，推理任务比检测任务（如毒性检测）退化更少，因为生成的解释提供了必要的上下文。我们相信Indi-RomCoM有助于社区开发包容性的多语言系统。
英文摘要：
Romanized Code Mixing (RCM), where bilingual speakers fluidly blend local languages with English in Roman script, has emerged as the dominant form of communication across multilingual communities. While Large Language Models (LLMs) perform strongly on monolingual and native-script benchmarks, their ability to follow instructions and reason over RCM-based content remains largely unexplored. To this end, we introduce the Indi-RomCoM benchmark for facilitating systematic evaluation on Indic Romanized Code-Mixed instructions. Our benchmark spans seven instruction-following tasks, four widely spoken Indic languages, and three controlled code-mixing intensity levels. We extensively evaluate a suite of LLMs covering proprietary, open-weight, and Indic-focused models under zero- and few-shot settings. LLMs consistently underperform on RCM instructions, with performance degrading as code-mixing density increases. Furthermore, reasoning tasks suffer less degradation than detection tasks (e.g., Toxicity) because the generated explanations offer necessary context. We believe Indi-RomCoM helps the community in developing inclusive multilingual systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
