RealCritic：面向语言模型批评的有效性驱动评估
RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques
AI总结：
本文提出RealCritic基准，采用闭环方法评估LLM批评能力，涵盖自我、交叉和迭代批评，发现经典模型在批评场景中显著落后于推理模型o1-mini。
摘要：
批评对于提升大型语言模型（LLMs）的性能至关重要，它通过识别缺陷并提出改进建议，使模型能够实现自我改进并为他人提供建设性反馈。然而，由于批评任务的开放性，评估LLMs的批评能力面临重大挑战。在本工作中，我们引入了一个新基准，旨在评估LLMs的批评能力。与现有基准通常以开环方式运行不同，我们的方法采用闭环方法论，评估由批评生成的修正质量。此外，该基准整合了自我批评、交叉批评和迭代批评等特性，这些特性对于区分高级推理模型与更经典模型的能力至关重要。我们使用八个具有挑战性的推理任务来实现该基准。我们有几个有趣的发现。首先，尽管在直接思维链生成中表现出相当的性能，经典LLMs在所有批评场景中显著落后于基于高级推理的模型o1-mini。其次，在自我批评和迭代批评设置中，经典LLMs甚至可能相对于其基线能力表现不佳。我们希望该基准能作为宝贵的资源，指导未来的进展。代码和数据可在\url{https://github.com/tangzhy/RealCritic}获取。
英文摘要：
Critiques are important for enhancing the performance of Large Language Models (LLMs), enabling both self-improvement and constructive feedback for others by identifying flaws and suggesting improvements. However, evaluating the critique capabilities of LLMs presents a significant challenge due to the open-ended nature of the task. In this work, we introduce a new benchmark designed to assess the critique capabilities of LLMs. Unlike existing benchmarks, which typically function in an open-loop fashion, our approach employs a closed-loop methodology that evaluates the quality of corrections generated from critiques. Moreover, the benchmark incorporates features such as self-critique, cross-critique, and iterative critique, which are crucial for distinguishing the abilities of advanced reasoning models from more classical ones. We implement this benchmark using eight challenging reasoning tasks. We have several interesting findings. First, despite demonstrating comparable performance in direct chain-of-thought generation, classical LLMs significantly lag behind the advanced reasoning-based model o1-mini across all critique scenarios. Second, in self-critique and iterative critique settings, classical LLMs may even underperform relative to their baseline capabilities. We hope that this benchmark will serve as a valuable resource to guide future advancements. The code and data are available at \url{https://github.com/tangzhy/RealCritic}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
