arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.05370cs.CL
收藏

挖掘还是合成?重新思考数学推理迭代对齐中的探索效率

Mining or Synthesis? Rethinking Exploration Efficiency in Iterative Alignment of Mathematical Reasoning

Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学深圳研究院);Huawei Large Model Data Technology Lab(华为大模型数据技术实验室);Huawei Multimodal Model Lab(华为多模态模型实验室);Department of Statistics and Data Science, Tsinghua University, Beijing, China(清华大学统计与数据科学系)

Jun Rao, Zixiong Yu, Xuebo Liu, Guhan Chen, Jing Li, Hejin Wang, Jiansheng Wei, Xiaojun Meng, Min Zhang

AI总结:

针对数学推理任务中迭代DPO对齐时高N采样收益递减且引入噪声的问题,提出PACE框架,通过低预算探索与纠错合成偏好对,以约1/5计算量达到或超越高N基线性能。

摘要:

迭代直接偏好优化(DPO)已成为在推理任务中对齐大语言模型的广泛使用的范式。现有方法通常依赖Best-of-N采样($N\geq8$)从分布尾部挖掘正轨迹。在这项工作中,我们表明在数学推理中,增加$N$会导致收益递减,同时增加验证器引起的假阳性风险和策略更新所需的数据分布偏移。为了解决这个问题,我们引入了PACE(通过纠错探索的近端对齐),一种基于生成的纠错框架,用低预算探索($2\leq N\leq3$)取代穷举挖掘。PACE不是搜索越来越稀有的正样本,而是通过纠错后见优化和验证引导过滤,从失败的探索中合成高保真偏好对。实验上,PACE匹配或超过了DPO-R1($N=16$)的性能,同时使用约$1/5$的计算量,并且在20%标签损坏下保持鲁棒,而高$N$基线表现出明显更高的噪声利用。

英文摘要:

Iterative Direct Preference Optimization (DPO) has emerged as a widely used paradigm for aligning Large Language Models on reasoning tasks. Existing approaches typically rely on Best-of-N sampling ($N\geq8$) to mine positive trajectories from the distribution tail. In this work, we show that in mathematical reasoning, increasing $N$ yields diminishing returns while increasing verifier-induced false-positive risk and the distribution shift required for policy updates. To address this, we introduce PACE (Proximal Alignment via Corrective Exploration), a generation-based corrective framework that replaces exhaustive mining with low-budget exploration ($2\leq N\leq3$). Rather than searching for increasingly rare positive samples, PACE synthesizes high-fidelity preference pairs from failed explorations through corrective hindsight refinement and verification-guided filtering. Empirically, PACE matches or exceeds the performance of DPO-R1 ($N=16$) while using about $1/5$ of the compute, and remains robust under 20\% label corruption, where high-$N$ baselines exhibit substantially higher noise exploitation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑