纠偏评估偏好：通过基于困惑度感知的强化学习提升大语言模型对数学推理的评判能力
Rectify Evaluation Preference: Improving LLMs' Critique on Math Reasoning via Perplexity-aware Reinforcement Learning
AI总结：
针对大语言模型评判数学推理时存在的失衡评估偏好问题，本文提出困惑度感知的强化学习算法，构建OPS基准验证了方法可有效提升模型的数学推理评判能力。
摘要：
为提升大语言模型（LLMs）的多步数学推理（MsMR）能力，通过自动评判MsMR推理过程中的错误并给出问题解答的最终判定，从语料中获取可扩展的监督信号至关重要。现有多数方法依赖构建高质量的监督微调示例来增强评判能力，鲜少深入探究LLMs评判性能不佳的根本原因。本文正交量化并研究了潜在原因——失衡的评估偏好，并开展了统计偏好分析。受该原因分析的启发，本文提出一种新颖的困惑度感知强化学习算法来纠偏评估偏好，提升评判能力。具体而言，为探究LLMs的评判特性，本文精心构建了一对多问题解答（OPS）基准，用于量化LLMs在评判自身与其他模型生成的问题解答时的行为差异。随后，为深入研究该行为差异，本文开展了面向困惑度的统计偏好分析，发现了一个有趣现象——“LLMs倾向于将困惑度更低的解答判定为正确”，该现象被命名为“失衡评估偏好”。为纠偏该偏好，本文将困惑度作为组相对策略优化（Group Relative Policy Optimization）算法中的指挥棒，支持LLMs探索将低困惑度解答判定为错误、高困惑度解答判定为正确的轨迹。在自建OPS基准与现有可用评判基准上的大量实验结果证明了本文方法的有效性。
英文摘要：
To improve Multi-step Mathematical Reasoning (MsMR) of Large Language Models (LLMs), it is crucial to obtain scalable supervision from the corpus by automatically critiquing mistakes in the reasoning process of MsMR and rendering a final verdict of the problem-solution. Most existing methods rely on crafting high-quality supervised fine-tuning demonstrations for critiquing capability enhancement and pay little attention to delving into the underlying reason for the poor critiquing performance of LLMs. In this paper, we orthogonally quantify and investigate the potential reason -- imbalanced evaluation preference, and conduct a statistical preference analysis. Motivated by the analysis of the reason, a novel perplexity-aware reinforcement learning algorithm is proposed to rectify the evaluation preference, elevating the critiquing capability. Specifically, to probe into LLMs' critiquing characteristics, a One-to-many Problem-Solution (OPS) benchmark is meticulously constructed to quantify the behavior difference of LLMs when evaluating the problem solutions generated by itself and others. Then, to investigate the behavior difference in depth, we conduct a statistical preference analysis oriented on perplexity and find an intriguing phenomenon -- ``LLMs incline to judge solutions with lower perplexity as correct'', which is dubbed as \textit{imbalanced evaluation preference}. To rectify this preference, we regard perplexity as the baton in the algorithm of Group Relative Policy Optimization, supporting the LLMs to explore trajectories that judge lower perplexity as wrong and higher perplexity as correct. Extensive experimental results on our built OPS and existing available critic benchmarks demonstrate the validity of our method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
