释放大语言模型的递归思维：通过精炼进行对齐
Unlocking Recursive Thinking of LLMs: Alignment via Refinement
AI总结：
针对LLM递归推理与专家蒸馏数据不足问题，提出AvR对齐方法，通过批评—改进精炼过程和可微奖励优化合成多轮长CoT数据，仅3k样本即显著提升LLaMA-3-8B-Instruct在AlpacaEval 2.0上的胜率。
摘要：
OpenAI o1系列模型已经表明，利用长形式思维链（Chain of Thought，CoT）可以显著提升性能。然而，大语言模型（Large Language Models，LLMs）的递归思维能力仍然有限，尤其是在缺乏专家整理数据用于蒸馏的情况下。本文提出 AvR：Alignment via Refinement（通过精炼进行对齐），这是一种旨在通过长形式CoT释放LLMs递归推理潜力的新方法。AvR引入了一个整合批评与改进动作的精炼过程，并以可微学习技术为指导来优化“精炼感知奖励”。因此，合成的多轮数据可以被组织成一条长的精炼思维，从而进一步实现测试时扩展。实验结果表明，AvR显著优于传统偏好优化方法。值得注意的是，仅使用3k个合成样本，我们的方法就在AlpacaEval 2.0上将LLaMA-3-8B-Instruct模型的胜率提升了20%以上。代码已发布于Github（https://github.com/Banner-Z/AvR.git）。
英文摘要：
The OpenAI o1-series models have demonstrated that leveraging long-form Chain of Thought (CoT) can substantially enhance performance. However, the recursive thinking capabilities of Large Language Models (LLMs) remain limited, particularly in the absence of expert-curated data for distillation. In this paper, we propose \textbf{AvR}: \textbf{Alignment via Refinement}, a novel method aimed at unlocking the potential of LLMs for recursive reasoning through long-form CoT. AvR introduces a refinement process that integrates criticism and improvement actions, guided by differentiable learning techniques to optimize \textbf{refinement-aware rewards}. As a result, the synthesized multi-round data can be organized as a long refinement thought, further enabling test-time scaling. Experimental results show that AvR significantly outperforms conventional preference optimization methods. Notably, with only 3k synthetic samples, our method boosts the performance of the LLaMA-3-8B-Instruct model by over 20\% in win rate on AlpacaEval 2.0. Our code is available at Github (https://github.com/Banner-Z/AvR.git).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
