MEMERAG：面向检索增强生成的多语言端到端元评估基准
MEMERAG: A Multilingual End-to-End Meta-Evaluation Benchmark for Retrieval Augmented Generation
AI总结：
本文提出多语言端到端元评估基准MEMERAG，基于MIRACL数据集和原生语言问题构建，用于评估RAG系统的忠实性与相关性，并验证其能可靠识别多语言自动评估器的改进。
摘要：
检索增强生成（RAG）系统的自动评估依赖于忠实性和相关性等细粒度维度，这些维度由专家人工标注者进行评判。元评估基准支持开发与人类判断高度相关的自动评估器。然而，现有基准主要关注英语或使用翻译数据，这无法捕捉文化细微差别。原生语言方法能更好地代表最终用户体验。在本工作中，我们开发了一个多语言端到端元评估RAG基准（MEMERAG）。我们的基准建立在流行的MIRACL数据集之上，使用原生语言问题，并利用多种大型语言模型（LLMs）生成响应，然后由专家标注者评估其忠实性和相关性。我们描述了标注过程，并表明其实现了较高的标注者间一致性。随后，我们根据人类评估者的评判，分析了生成答案的LLMs在不同语言上的表现。最后，我们将该数据集应用于我们的主要用例，即对多语言自动评估器（LLM-as-a-judge）进行基准测试。我们表明，我们的基准能够可靠地识别先进提示技术和LLMs带来的改进。我们的数据集可在https://github.com/amazon-science/MEMERAG获取。
英文摘要：
Automatic evaluation of retrieval augmented generation (RAG) systems relies on fine-grained dimensions like faithfulness and relevance, as judged by expert human annotators. Meta-evaluation benchmarks support the development of automatic evaluators that correlate well with human judgement. However, existing benchmarks predominantly focus on English or use translated data, which fails to capture cultural nuances. A native approach provides a better representation of the end user experience. In this work, we develop a Multilingual End-to-end Meta-Evaluation RAG benchmark (MEMERAG). Our benchmark builds on the popular MIRACL dataset, using native-language questions and generating responses with diverse large language models (LLMs), which are then assessed by expert annotators for faithfulness and relevance. We describe our annotation process and show that it achieves high inter-annotator agreement. We then analyse the performance of the answer-generating LLMs across languages as per the human evaluators. Finally we apply the dataset to our main use-case which is to benchmark multilingual automatic evaluators (LLM-as-a-judge). We show that our benchmark can reliably identify improvements offered by advanced prompting techniques and LLMs. Our dataset is available at https://github.com/amazon-science/MEMERAG
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
