Auto-PRE：一种用于语言生成评估的自动化且成本高效的同行评审框架
Auto-PRE: An Automatic and Cost-Efficient Peer-Review Framework for Language Generation Evaluation
AI总结：
针对大语言模型传统评估方法成本高、依赖人工参考等问题，提出基于同行评审的Auto-PRE框架，按三大特质自动遴选评审LLM，在三类任务上达最优性能且降本，为LLM评审自动化提供参考。
摘要：
大语言模型（LLM）的快速发展凸显了对高效可靠的性能评估方法的需求。传统评估方法常面临成本高、任务格式受限、依赖人工参考文本以及存在系统性偏差等挑战。为解决这些局限，我们提出Auto-PRE——一种受同行评审流程启发的自动化LLM评估框架。与以往依赖人工标注的方法不同，Auto-PRE基于一致性、相关性和自信心三大核心特质自动遴选评审用LLM，这三项特质分别对应指令阶段、内容阶段和响应阶段，共同覆盖整个评估流程。在摘要生成、非事实类问答、对话生成三项代表性任务上的实验表明，Auto-PRE达到了当前最优（state-of-the-art）性能，同时显著降低了评估成本。此外，我们提出的自动化资质考核框架具备结构化与可扩展的设计，为实现“LLM作为评审者”的自动化评估提供了宝贵洞见，也为更先进的基于LLM的评估框架铺平了道路。
英文摘要：
The rapid development of large language models (LLMs) has highlighted the need for efficient and reliable methods to evaluate their performance. Traditional evaluation methods often face challenges like high costs, limited task formats, dependence on human references, and systematic biases. To address these limitations, we propose Auto-PRE, an automatic LLM evaluation framework inspired by the peer review process. Unlike previous approaches that rely on human annotations, Auto-PRE automatically selects evaluator LLMs based on three core traits: consistency, pertinence, and self-confidence, which correspond to the instruction, content, and response stages, respectively, and collectively cover the entire evaluation process. Experiments on three representative tasks, including summarization, non-factoid QA, and dialogue generation, demonstrate that Auto-PRE achieves state-of-the-art performance while significantly reducing evaluation costs. Furthermore, the structured and scalable design of our automatic qualification exam framework provides valuable insights into automating the evaluation of LLMs-as-judges, paving the way for more advanced LLM-based evaluation frameworks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
