CriticBench：评估LLM批判-修正推理的基准
CriticBench: Benchmarking LLMs for Critique-Correct Reasoning
AI总结：
本文提出CriticBench基准，评估17个LLM在生成、批判和修正推理（GQC）上的能力，发现批判训练显著提升性能，修正效果因任务而异，且模型间批判存在强弱互补的动态关系。
摘要：
大型语言模型（LLMs）批判和修正其推理的能力对于其在评估、反馈提供和自我改进中的应用至关重要。本文介绍了CriticBench，这是一个全面的基准，旨在评估LLMs在各种任务中批判和纠正其推理的能力。CriticBench包含五个推理领域：数学、常识、符号、编程和算法。它汇编了15个数据集，并整合了来自三个LLM系列的响应。利用CriticBench，我们评估并剖析了17个LLM在生成、批判和修正推理方面的表现，即GQC推理。我们的发现揭示：（1）GQC能力呈线性关系，以批判为中心的训练显著提升了性能；（2）修正有效性存在任务依赖性差异，面向逻辑的任务更易于修正；（3）GQC知识不一致性随着模型规模增加而减少；（4）一种有趣的模型间批判动态，即较强模型更擅长批判较弱模型，而较弱模型在自我批判中竟能超越较强模型。我们希望这些关于LLMs细致批判-修正推理的见解能促进LLM批判和自我改进的进一步研究。
英文摘要：
The ability of Large Language Models (LLMs) to critique and refine their reasoning is crucial for their application in evaluation, feedback provision, and self-improvement. This paper introduces CriticBench, a comprehensive benchmark designed to assess LLMs' abilities to critique and rectify their reasoning across a variety of tasks. CriticBench encompasses five reasoning domains: mathematical, commonsense, symbolic, coding, and algorithmic. It compiles 15 datasets and incorporates responses from three LLM families. Utilizing CriticBench, we evaluate and dissect the performance of 17 LLMs in generation, critique, and correction reasoning, i.e., GQC reasoning. Our findings reveal: (1) a linear relationship in GQC capabilities, with critique-focused training markedly enhancing performance; (2) a task-dependent variation in correction effectiveness, with logic-oriented tasks being more amenable to correction; (3) GQC knowledge inconsistencies that decrease as model size increases; and (4) an intriguing inter-model critiquing dynamic, where stronger models are better at critiquing weaker ones, while weaker models can surprisingly surpass stronger ones in their self-critique. We hope these insights into the nuanced critique-correct reasoning of LLMs will foster further research in LLM critique and self-improvement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
