arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.07641cs.CLcs.LG
收藏

评估大型语言模型在指令遵循评价方面的表现

Evaluating Large Language Models at Evaluating Instruction Following

Tsinghua University(清华大学);Princeton University(普林斯顿大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, Danqi Chen

AI总结:

本文提出 LLMBar 元评价基准,通过 419 对具有迷惑性的指令遵循/偏离输出检验 LLM 评价器,并给出提示策略以缩小其与人类评价的差距。

摘要:

随着大型语言模型(LLM)研究持续加速,基于 LLM 的评价已成为人工评价的一种可扩展且具有成本效益的替代方案,用于比较不断增加的模型列表。本文研究这些“LLM 评价器”的有效性,尤其是使用它们评估指令遵循,即衡量生成文本在多大程度上遵守给定指令的指标。我们引入一个具有挑战性的元评价基准 LLMBar,旨在测试 LLM 评价器辨别指令遵循输出的能力。作者人工整理了 419 对输出,其中一个遵循指令,另一个偏离指令,但后者可能具有误导 LLM 评价器的欺骗性特征,例如更具吸引力的语气。与现有元评价相反,我们发现不同评价器(即 LLM 与提示词的组合)在 LLMBar 上表现出明显不同的性能,即使得分最高的评价器也有相当大的改进空间。我们还提出了一套新颖的提示策略,进一步缩小 LLM 评价器与人类评价者之间的差距。借助 LLMBar,我们希望更深入地认识 LLM 评价器,并推动未来研究开发更好的指令遵循模型。

英文摘要:

As research in large language models (LLMs) continues to accelerate, LLM-based evaluation has emerged as a scalable and cost-effective alternative to human evaluations for comparing the ever increasing list of models. This paper investigates the efficacy of these ``LLM evaluators'', particularly in using them to assess instruction following, a metric that gauges how closely generated text adheres to the given instruction. We introduce a challenging meta-evaluation benchmark, LLMBar, designed to test the ability of an LLM evaluator in discerning instruction-following outputs. The authors manually curated 419 pairs of outputs, one adhering to instructions while the other diverging, yet may possess deceptive qualities that mislead an LLM evaluator, e.g., a more engaging tone. Contrary to existing meta-evaluation, we discover that different evaluators (i.e., combinations of LLMs and prompts) exhibit distinct performance on LLMBar and even the highest-scoring ones have substantial room for improvement. We also present a novel suite of prompting strategies that further close the gap between LLM and human evaluators. With LLMBar, we hope to offer more insight into LLM evaluators and foster future research in developing better instruction-following models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑