arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.07889cs.CL
收藏

LLM 比较评估:通过大型语言模型进行成对比较的零样本自然语言生成评估

LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models

University of Cambridge(剑桥大学)

Adian Liusie, Potsawee Manakul, Mark J. F. Gales

AI总结:

本文提出利用大型语言模型进行零样本自然语言生成评估,通过成对比较优于绝对评分,并针对位置偏差提出去偏方法,实现高效且竞争力强的评估。

摘要:

当前大型语言模型(LLM)的发展已在各种自然语言任务中展现出令人印象深刻的零样本能力。这些系统的一个有趣应用是在自然语言生成(NLG)的自动评估中,这是一个极具挑战性且具有巨大实际益处的领域。在本文中,我们探讨了利用LLM的涌现能力进行零样本NLG评估的两种选项:绝对分数预测,以及使用候选对之间相对比较的比较评估。尽管比较评估在NLG评估中尚未得到广泛研究,但我们注意到,人类通常发现比较两个选项比独立评分每个选项更直观。这项工作从多个角度审视了比较评估:与绝对评分相比的性能;提示中的位置偏差;以及在比较次数方面的高效排序。我们表明,LLM比较评估是一种简单、通用且有效的NLG评估方法。对于中等规模的开源LLM,如FlanT5和Llama2-chat,比较评估优于提示评分,并且在许多情况下可以达到与最先进方法竞争的性能。此外,我们证明LLM在进行成对比较时常常表现出强烈的位置偏差,并提出了可以进一步提高性能的去偏方法。

英文摘要:

Current developments in large language models (LLMs) have enabled impressive zero-shot capabilities across various natural language tasks. An interesting application of these systems is in the automated assessment of natural language generation (NLG), a highly challenging area with great practical benefit. In this paper, we explore two options for exploiting the emergent abilities of LLMs for zero-shot NLG assessment: absolute score prediction, and comparative assessment which uses relative comparisons between pairs of candidates. Though comparative assessment has not been extensively studied in NLG assessment, we note that humans often find it more intuitive to compare two options rather than scoring each one independently. This work examines comparative assessment from multiple perspectives: performance compared to absolute grading; positional biases in the prompt; and efficient ranking in terms of the number of comparisons. We illustrate that LLM comparative assessment is a simple, general and effective approach for NLG assessment. For moderate-sized open-source LLMs, such as FlanT5 and Llama2-chat, comparative assessment is superior to prompt scoring, and in many cases can achieve performance competitive with state-of-the-art methods. Additionally, we demonstrate that LLMs often exhibit strong positional biases when making pairwise comparisons, and we propose debiasing methods that can further improve performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑