对于大型语言模型的多语言指令微调，这是优质数据还是仅仅糟糕的多语言评估？
Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models?
AI总结：
本研究探讨多语言大模型微调与评估中过度依赖翻译带来的问题，发现母语或生成基准能揭示母语与翻译指令数据的显著差异，并证明正则化有助于在结构化任务上弥合此差距。
摘要：
多语言大型语言模型被设计、声称并期望服务于不同语言的使用者。我们假设，由于严重依赖翻译，当前微调和评估这些模型的实践可能无法完全实现这一目标，因为翻译无法覆盖特定语言的知识，且会引入翻译缺陷。目前尚不清楚指令数据的性质是否会影响模型输出；反过来，翻译后的测试集是否能捕捉到这些细微差别也值得怀疑。由于在两个阶段使用翻译数据的做法常常是耦合的，这些缺陷可能一直被忽视。本研究通过在指令微调和评估阶段使用受控的母语或翻译数据来调查这些问题。我们发现，母语或生成基准能揭示母语与翻译指令数据之间的显著差异，尤其是在模型性能较高时，而其他类型的测试集则无法做到。往返翻译与单次翻译的比较反映了母语资源知识的重要性。最后，我们证明正则化有助于在结构化任务上弥合这一差距，但在生成任务上则不行。
英文摘要：
Multilingual large language models are designed, claimed, and expected to cater to speakers of varied languages. We hypothesise that the current practices of fine-tuning and evaluating these models may not perfectly align with this objective owing to a heavy reliance on translation, which cannot cover language-specific knowledge but can introduce translation defects. It remains unknown whether the nature of the instruction data has an impact on the model output; conversely, it is questionable whether translated test sets can capture such nuances. Due to the often coupled practices of using translated data in both stages, such imperfections could have been overlooked. This work investigates these issues using controlled native or translated data during the instruction tuning and evaluation stages. We show that native or generation benchmarks reveal a notable difference between native and translated instruction data especially when model performance is high, whereas other types of test sets cannot. The comparison between round-trip and single-pass translations reflects the importance of knowledge from language-native resources. Finally, we demonstrate that regularization is beneficial to bridging this gap on structured but not generative tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
