arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.15329cs.CL
收藏

DnA-Eval: 基于分解与聚合的大语言模型评估增强方法

DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation

National University of Singapore(新加坡国立大学);Salesforce Research(Salesforce研究院);Nanyang Technological University(南洋理工大学);Institute for Infocomm Research (IR)(信息通信研究院);A*STAR(新加坡科技研究局)

Minzhi Li, Zhengyuan Liu, Shumin Deng, Shafiq Joty, Nancy F. Chen, Min-Yen Kan

AI总结:

针对大语言模型作为文本评估器缺乏可解释性的问题,提出基于教学实践的分解聚合评估框架DnA-Eval,既提升评估可解释性,又在多种元评估基准上最高提升39.6%的评估性能。

摘要:

大语言模型(LLMs)研究的快速发展为生成文本的评估开辟了新可能。LLMs可作为可扩展且经济的评估器,但这类评估器的可靠性问题已成为关键研究课题。此前针对LLM作为评审的元评估研究,仅通过单次提示LLM得出最终评估结论,再计算LLM输出与人类标注的一致性,这使得LLM评估能力的解读缺乏可解释性。针对这一挑战,我们提出“分解与聚合(Decompose and Aggregate)”方法,依据教学实践将评估流程拆分为不同阶段。实验表明,该方法不仅为理解LLM的评估效果提供了更具可解释性的视角,还在多种元评估基准上,使不同LLM的评估性能最高提升39.6%。

英文摘要:

The acceleration of Large Language Models (LLMs) research has opened up new possibilities for evaluating generated texts. They serve as scalable and economical evaluators, but the question of how reliable these evaluators are has emerged as a crucial research question. Prior research efforts in the meta-evaluation of LLMs as judges limit the prompting of an LLM to a single use to obtain a final evaluation decision. They then compute the agreement between LLMs' outputs and human labels. This lacks interpretability in understanding the evaluation capability of LLMs. In light of this challenge, we propose Decompose and Aggregate, which breaks down the evaluation process into different stages based on pedagogical practices. Our experiments illustrate that it not only provides a more interpretable window for how well LLMs evaluate, but also leads to improvements up to 39.6% for different LLMs on a variety of meta-evaluation benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑