arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.01383cs.CL
收藏

基于LLM的NLG评估:现状与挑战

LLM-based NLG Evaluation: Current Status and Challenges

Peking University(北京大学)

Mingqi Gao, Xinyu Hu, Jie Ruan, Xiao Pu, Xiaojun Wan

AI总结:

本文综述基于LLM的NLG自动评估方法,对LLM指标、提示、微调及人机协同评估分类并分析优劣,指出开放问题与未来方向。

摘要:

评估自然语言生成(NLG)是自然语言处理中一个至关重要但颇具挑战性的问题。主要衡量系统输出与参考文本之间内容(例如n-gram)重叠的传统评估指标远不能令人满意,而近年来ChatGPT等大型语言模型(LLM)在NLG评估中展现出巨大潜力。研究者已提出多种基于LLM的自动评估方法,包括源自LLM的指标、提示LLM、微调LLM以及人类与LLM协同评估。在本综述中,我们首先对基于LLM的NLG评估方法进行分类,并分别讨论其优缺点。最后,我们讨论该领域的若干开放问题,并指出未来的研究方向。

英文摘要:

Evaluating natural language generation (NLG) is a vital but challenging problem in natural language processing. Traditional evaluation metrics mainly capturing content (e.g. n-gram) overlap between system outputs and references are far from satisfactory, and large language models (LLMs) such as ChatGPT have demonstrated great potential in NLG evaluation in recent years. Various automatic evaluation methods based on LLMs have been proposed, including metrics derived from LLMs, prompting LLMs, fine-tuning LLMs, and human-LLM collaborative evaluation. In this survey, we first give a taxonomy of LLM-based NLG evaluation methods, and discuss their pros and cons, respectively. Lastly, we discuss several open problems in this area and point out future research directions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑