arXiv 2501.057272025-08-04cs.CLcs.AIcs.LG

大型语言模型中的自我进化批判能力

Self-Evolving Critique Abilities in Large Language Models

The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Shenzhen International Center for Industrial and Applied Mathematics（深圳国际工业与应用数学中心）；Qwen Team, Alibaba Inc.（阿里云通义团队）

Zhengyang Tang, Ziniu Li, Zhenyang Xiao, Tian Ding, Ruoyu Sun, Benyou Wang, Dayiheng Liu, Fei Huang, Tianyu Liu, Bowen Yu, Junyang Lin

备注：: Accepted by COLM 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出SCRIT框架，利用自生成数据和对比批评方法，无需外部监督即可增强LLMs的批判能力，在数学和科学推理基准上显著提升批评纠正准确率和错误识别F1分数。

摘要：

尽管大型语言模型（LLMs）表现出色，但它们面临一个关键挑战：在人类评估困难或LLMs可能超越人类的任务中提供反馈。在这种情况下，利用LLMs自身的批判能力——识别和纠正缺陷——显示出相当大的前景。本文探讨了增强LLMs的批判能力，指出当前方法依赖于人工标注或更强大的模型，使得在没有外部监督的情况下提高批判能力的挑战仍未解决。我们引入了SCRIT（自我进化批评家），一个使用自生成数据训练LLMs以进化其批判能力的框架。为了解决朴素生成数据质量低的问题，我们提出了一种对比批评方法，在数据合成过程中使用参考解决方案来增强模型对关键概念的理解，并结合自我验证方案以确保数据质量。最终训练的模型在推理时无需任何参考解决方案。使用领先的LLM Qwen2.5-72B-Instruct实现，SCRIT在涵盖数学和科学推理的广泛基准测试中展示了一致的改进：在批评纠正准确率上实现了10.0%的相对提升，在错误识别F1分数上实现了19.0%的相对改进。我们的分析表明，SCRIT的性能随数据和模型规模正向扩展，并通过多轮迭代实现持续改进。

英文摘要：

Despite their remarkable performance, Large Language Models (LLMs) face a critical challenge: providing feedback for tasks where human evaluation is difficult or where LLMs potentially outperform humans. In such scenarios, leveraging the critique ability of LLMs themselves - identifying and correcting flaws - shows considerable promise. This paper explores enhancing critique abilities of LLMs, noting that current approaches rely on human annotations or more powerful models, leaving the challenge of improving critique abilities without external supervision unresolved. We introduce SCRIT (Self-evolving CRITic), a framework that trains LLMs with self-generated data to evolve their critique abilities. To address the low quality of naively generated data, we propose a contrastive-critic approach that uses reference solutions during data synthesis to enhance the model's understanding of key concepts, and incorporates a self-validation scheme to ensure data quality. The final trained model operates without any reference solutions at inference time. Implemented with Qwen2.5-72B-Instruct, a leading LLM, SCRIT demonstrates consistent improvements across a wide range of benchmarks spanning both mathematical and scientific reasoning: achieving a 10.0\% relative gain in critique-correction accuracy and a 19.0\% relative improvement in error identification F1-score. Our analysis reveals that SCRIT's performance scales positively with data and model size and enables continuous improvement through multi-round iterations.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。