arXiv 2410.133942025-07-18cs.CL

面向多语言大语言模型评估的跨语言自动评测

Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs

Nilekani Centre at AI4Bharat（AI4Bharat 奈尔肯中心）；IIT Madras（印度理工学院马德拉斯分校）；New York University（纽约大学）；Microsoft（微软）；National Institute of Information and Communications Technology, Kyoto, Japan（京都国立信息与通信技术研究所）

Sumanth Doddapaneni, Mohammed Safi Ur Rahman Khan, Dilip Venkatesh, Raj Dabre, Anoop Kunchukuttan, Mitesh M. Khapra

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出跨语言自动评测套件CIA Suite，包括评估模型Hercule和测试集Recon，通过利用英语参考答案实现低资源多语言评估，实验表明其与人工判断一致性优于专有模型。

摘要：

评估机器生成文本在自然语言处理中仍然是一个重大挑战，尤其是对于非英语语言。当前的方法，包括自动指标、人工评估和基于大语言模型的评估，主要集中于英语，这揭示了多语言评估框架中的显著空白。我们提出了跨语言自动评测套件（CIA Suite），这是一个可扩展的框架，包含评估器大语言模型（Hercule）和一个专门为多语言评估设计的新型测试集（Recon）。我们的测试集包含500条人工标注的指令，涵盖多种任务能力，并配有六种语言的人工判断分数。这将有助于对通用多语言大语言模型进行基准测试，并促进评估器大语言模型的元评估。所提出的模型Hercule是一种跨语言评估模型，通过学习基于容易获得的英语参考答案为响应分配分数，来解决目标语言中参考答案稀缺的问题。我们的实验表明，与专有模型相比，Hercule与人工判断的一致性更高，证明了这种跨语言评估在低资源场景中的有效性。此外，它在未见语言的零样本评估中同样有效。本研究是首次使用大语言模型对跨语言评估进行全面考察，为多语言评估提供了一种可扩展且有效的方法。所有代码、数据集和模型都将公开，以推动这一重要领域的进一步研究。

英文摘要：

Evaluating machine-generated text remains a significant challenge in NLP, especially for non-English languages. Current methodologies, including automated metrics, human assessments, and LLM-based evaluations, predominantly focus on English, revealing a significant gap in multilingual evaluation frameworks. We introduce the Cross Lingual Auto Evaluation (CIA) Suite, an extensible framework that includes evaluator LLMs (Hercule) and a novel test set (Recon) specifically designed for multilingual evaluation. Our test set features 500 human-annotated instructions spanning various task capabilities along with human judgment scores across six languages. This would enable benchmarking of general-purpose multilingual LLMs and facilitate meta-evaluation of Evaluator LLMs. The proposed model, Hercule, is a cross-lingual evaluation model that addresses the scarcity of reference answers in the target language by learning to assign scores to responses based on easily available reference answers in English. Our experiments demonstrate that Hercule aligns more closely with human judgments compared to proprietary models, demonstrating the effectiveness of such cross-lingual evaluation in low resource scenarios. Further, it is also effective in zero-shot evaluation on unseen languages. This study is the first comprehensive examination of cross-lingual evaluation using LLMs, presenting a scalable and effective approach for multilingual assessment. All code, datasets, and models will be publicly available to enable further research in this important area.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。