探索基于LLM的评估器在多语言NLG评估中的能力
Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluators
AI总结：
本研究通过相关性分析、扰动攻击和微调，全面分析了10个LLM在多语言NLG评估中的表现，发现其评估能力在高资源与低资源语言间存在显著不平衡，建议低资源语言场景应获更多关注。
摘要：
先前研究表明，LLM在多语言NLG评估任务中具有潜力。然而，现有研究尚未充分探讨LLM在不同语言间评估能力的差异。为此，本研究通过相关性分析、扰动攻击和微调，对10个近期LLM的多语言评估表现进行了全面分析，涵盖高资源语言和低资源语言。我们发现：1）在提示中排除参考答案并使用大参数的基于LLM的评估器，能够在各种语言中带来更好的性能；2）大多数基于LLM的评估器在高资源语言中与人类判断的相关性高于低资源语言；3）在对这些攻击最敏感的语言中，它们也往往与人类判断的相关性最高；4）使用特定语言的数据进行微调，能在多种语言中为模型的评估性能带来广泛一致的提升。我们的发现突显了LLM在不同语言间评估能力的不平衡，并建议低资源语言场景值得更多关注。
英文摘要：
Previous research has shown that LLMs have potential in multilingual NLG evaluation tasks. However, existing research has not fully explored the differences in the evaluation capabilities of LLMs across different languages. To this end, this study provides a comprehensive analysis of the multilingual evaluation performance of 10 recent LLMs, spanning high-resource and low-resource languages through correlation analysis, perturbation attacks, and fine-tuning. We found that 1) excluding the reference answer from the prompt and using large-parameter LLM-based evaluators leads to better performance across various languages; 2) most LLM-based evaluators show a higher correlation with human judgments in high-resource languages than in low-resource languages; 3) in the languages where they are most sensitive to such attacks, they also tend to exhibit the highest correlation with human judgments; and 4) fine-tuning with data from a particular language yields a broadly consistent enhancement in the model's evaluation performance across diverse languages. Our findings highlight the imbalance in LLMs'evaluation capabilities across different languages and suggest that low-resource language scenarios deserve more attention.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
