arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.15641cs.IRcs.CL
收藏

PRE:一个基于同行评审的大型语言模型评估器

PRE: A Peer Review Based Large Language Model Evaluator

Tsinghua University(清华大学);Zhongguancun Laboratory(中关村实验室)

Zhumin Chu, Qingyao Ai, Yiteng Tu, Haitao Li, Yiqun Liu

AI总结:

针对现有LLM评估方法成本高、泛化性低且存在偏见的问题,本文提出基于同行评审机制的自动评估框架PRE,通过资格考试选拔评审员模型并对被评估模型进行评分,实验证明其优于基线并有效减少了单一模型评估的偏见。

摘要:

大型语言模型(LLMs)的出色表现引起了学术界和工业界的极大关注。除了如何构建和训练LLMs之外,如何有效地评估和比较LLMs的能力也被公认为是一个重要且困难的问题。现有范式依赖于人工标注者或基于模型的评估器来评估LLMs在不同任务上的表现。然而,这些范式在实践中常常面临高成本、低泛化性和固有偏见等问题,使其无法长期支持LLMs的可持续发展。为了解决这些问题,受学术出版过程中广泛使用的同行评审系统启发,我们提出了一个新颖的框架,可以通过同行评审过程自动评估LLMs。具体而言,对于特定任务的评估,我们首先构建一个小型资格考试,从几个强大的LLMs中选拔“评审员”。然后,为了实际评估不同候选LLMs(即被评估者)撰写的“投稿”,我们使用评审员LLMs对投稿进行评分或比较。被评估LLMs的最终排名基于所有评审员提供的结果生成。我们在文本摘要任务上使用包括GPT-4在内的11个LLMs进行了广泛实验。结果表明,使用单个LLM进行评估时存在偏见。同时,我们的PRE模型优于所有基线,证明了同行评审机制的有效性。

英文摘要:

The impressive performance of large language models (LLMs) has attracted considerable attention from the academic and industrial communities. Besides how to construct and train LLMs, how to effectively evaluate and compare the capacity of LLMs has also been well recognized as an important yet difficult problem. Existing paradigms rely on either human annotators or model-based evaluators to evaluate the performance of LLMs on different tasks. However, these paradigms often suffer from high cost, low generalizability, and inherited biases in practice, which make them incapable of supporting the sustainable development of LLMs in long term. In order to address these issues, inspired by the peer review systems widely used in academic publication process, we propose a novel framework that can automatically evaluate LLMs through a peer-review process. Specifically, for the evaluation of a specific task, we first construct a small qualification exam to select "reviewers" from a couple of powerful LLMs. Then, to actually evaluate the "submissions" written by different candidate LLMs, i.e., the evaluatees, we use the reviewer LLMs to rate or compare the submissions. The final ranking of evaluatee LLMs is generated based on the results provided by all reviewers. We conducted extensive experiments on text summarization tasks with eleven LLMs including GPT-4. The results demonstrate the existence of biasness when evaluating using a single LLM. Also, our PRE model outperforms all the baselines, illustrating the effectiveness of the peer review mechanism.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑