Judge as A Judge：通过大语言模型的法官一致性改进检索增强生成的评估
Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models
AI总结：
本文提出ConsJudge方法，通过让大语言模型基于不同判断维度组合生成判断并利用法官一致性筛选，以提升对检索增强生成模型输出的评估准确性。
摘要：
检索增强生成（RAG）已被证明在缓解大语言模型（LLMs）的幻觉方面是有效的。然而，现有的自动评估指标无法公平地评估RAG模型在训练和评估过程中生成的输出。基于LLM的评判模型有潜力产生高质量的判断，但它们对评估提示高度敏感，导致在评判RAG模型输出时出现不一致。本文介绍了法官一致性（ConsJudge）方法，旨在增强LLMs为RAG模型生成更准确的评估。具体来说，ConsJudge提示LLMs基于判断维度的不同组合生成不同的判断，利用法官一致性来评估这些判断，并选择接受和拒绝的判断用于DPO训练。我们的实验表明，ConsJudge能够有效地为各种RAG模型和数据集优化提供更准确的判断。进一步分析显示，ConsJudge生成的判断与更优的LLM具有高度一致性。所有代码可在https://github.com/OpenBMB/ConsJudge获取。
英文摘要：
Retrieval-Augmented Generation (RAG) has proven its effectiveness in alleviating hallucinations for Large Language Models (LLMs). However, existing automated evaluation metrics cannot fairly evaluate the outputs generated by RAG models during training and evaluation. LLM-based judgment models provide the potential to produce high-quality judgments, but they are highly sensitive to evaluation prompts, leading to inconsistencies when judging the output of RAG models. This paper introduces the Judge-Consistency (ConsJudge) method, which aims to enhance LLMs to generate more accurate evaluations for RAG models. Specifically, ConsJudge prompts LLMs to generate different judgments based on various combinations of judgment dimensions, utilize the judge-consistency to evaluate these judgments and select the accepted and rejected judgments for DPO training. Our experiments show that ConsJudge can effectively provide more accurate judgments for optimizing RAG models across various RAG models and datasets. Further analysis reveals that judgments generated by ConsJudge have a high agreement with the superior LLM. All codes are available at https://github.com/OpenBMB/ConsJudge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
