arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.09302cs.LGcs.AIcs.CL
收藏

通过直接Q函数优化增强语言模型的多步推理能力

Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization

University of California, Los Angeles(加利福尼亚大学洛杉矶分校);ByteDance(字节跳动);Oregon State University(俄勒冈州立大学)

Kaixuan Ji, Guanlin Liu, Ning Dai, Qingping Yang, Renjie Zheng, Zheng Wu, Chen Dun, Quanquan Gu, Lin Yan

AI总结:

针对现有强化学习方法在语言模型多步推理任务中存在的计算资源消耗大或性能不足问题,提出基于MDP构建和SAC框架的DQO方法,在GSM8K和MATH数据集上优于现有方法,为语言模型对齐提供了高效的离线强化学习方案。

摘要:

强化学习(RL)在将大型语言模型(LLMs)与人类偏好对齐以及提升其执行复杂任务的能力方面发挥着关键作用。然而,当前方法要么因使用多个模型和大量在线采样训练而需要大量计算资源(例如PPO),要么被构建为多臂老虎机问题(例如DPO、DRO),这些方法在处理多步推理任务(如数学问题求解和涉及长思维链的复杂推理)时往往存在困难。为克服这些局限性,我们提出直接Q函数优化(DQO),将响应生成过程构建为马尔可夫决策过程(MDP),并利用软演员-评论家(SAC)框架直接优化由语言模型参数化的Q函数。DQO的MDP构建相比基于老虎机的方法具有结构优势,能实现更有效的过程监督。在GSM8K和MATH两个数学问题求解数据集上的实验结果表明,DQO优于现有方法,是一种很有前景的语言模型对齐离线强化学习方法。

英文摘要:

Reinforcement Learning (RL) plays a crucial role in aligning large language models (LLMs) with human preferences and improving their ability to perform complex tasks. However, current approaches either require significant computational resources due to the use of multiple models and extensive online sampling for training (e.g., PPO) or are framed as bandit problems (e.g., DPO, DRO), which often struggle with multi-step reasoning tasks, such as math problem solving and complex reasoning that involve long chains of thought. To overcome these limitations, we introduce Direct Q-function Optimization (DQO), which formulates the response generation process as a Markov Decision Process (MDP) and utilizes the soft actor-critic (SAC) framework to optimize a Q-function directly parameterized by the language model. The MDP formulation of DQO offers structural advantages over bandit-based methods, enabling more effective process supervision. Experimental results on two math problem-solving datasets, GSM8K and MATH, demonstrate that DQO outperforms previous methods, establishing it as a promising offline reinforcement learning approach for aligning language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑