Direct Value Optimization：利用精炼价值提升大语言模型的思维链推理能力
Direct Value Optimization: Improving Chain-of-Thought Reasoning in LLMs with Refined Values
AI总结：
提出强化学习框架 DVO，用单步价值信号和均方误差损失优化 LLM 思维链推理，无需偏好标注，并在数学与常识任务上以更少训练步骤超越离线偏好优化方法。
摘要：
我们提出 Direct Value Optimization（DVO，直接价值优化），这是一种创新的强化学习框架，用于增强大语言模型在复杂推理任务中的表现。不同于依赖偏好标签的传统方法，DVO 在各个单独推理步骤上利用价值信号，通过均方误差损失优化模型。DVO 的关键优势在于其细粒度监督，从而避免了对劳动密集型人工标注的需求。DVO 中的目标价值使用 Monte Carlo Tree Search（蒙特卡洛树搜索）或结果价值模型进行估计。我们在数学推理和常识推理任务上的实证分析表明，即使训练步骤更少，DVO 也始终优于现有的离线偏好优化技术。这些发现强调了价值信号在推进推理能力方面的重要性，并凸显出在缺乏显式人类偏好信息的场景下，DVO 是一种更优越的方法。
英文摘要：
We introduce Direct Value Optimization (DVO), an innovative reinforcement learning framework for enhancing large language models in complex reasoning tasks. Unlike traditional methods relying on preference labels, DVO utilizes value signals at individual reasoning steps, optimizing models via a mean squared error loss. The key benefit of DVO lies in its fine-grained supervision, circumventing the need for labor-intensive human annotations. Target values within the DVO are estimated using either Monte Carlo Tree Search or an outcome value model. Our empirical analysis on both mathematical and commonsense reasoning tasks shows that DVO consistently outperforms existing offline preference optimization techniques, even with fewer training steps. These findings underscore the importance of value signals in advancing reasoning capabilities and highlight DVO as a superior methodology under scenarios lacking explicit human preference information.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
