Reinforcement Learning (RL) plays a crucial role in aligning large language models (LLMs) with human preferences and improving their ability to perform complex tasks. However, current approaches either require significant computational resources due to the use of multiple models and extensive online sampling for training (e.g., PPO) or are framed as bandit problems (e.g., DPO, DRO), which often struggle with multi-step reasoning tasks, such as math problem solving and complex reasoning that involve long chains of thought. To overcome these limitations, we introduce Direct Q-function Optimization (DQO), which formulates the response generation process as a Markov Decision Process (MDP) and utilizes the soft actor-critic (SAC) framework to optimize a Q-function directly parameterized by the language model. The MDP formulation of DQO offers structural advantages over bandit-based methods, enabling more effective process supervision. Experimental results on two math problem-solving datasets, GSM8K and MATH, demonstrate that DQO outperforms previous methods, establishing it as a promising offline reinforcement learning approach for aligning language models.
A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning
Institute for Artificial Intelligence, Tsinghua University(清华大学人工智能研究院);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Tencent AI Lab(腾讯AI实验室)
Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMs
University of Leeds(利兹大学);The Alan Turing Institute(艾伦·图灵研究所);Tongji University(同济大学);Universitat Politècnica de València(瓦伦西亚理工大学);ValGRAI;Leverhulme Centre for the Future of Intelligence(利弗休姆未来智能中心)
GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents
Olin College of Engineering(奥林工程学院);University of Texas at Dallas(德克萨斯大学达拉斯分校);University of Oxford(牛津大学);University College London(伦敦大学学院);ShanghaiTech University(上海科技大学);Columbia University(哥伦比亚大学);University of Manchester(曼彻斯特大学)
Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Critic
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室);University of Chinese Academy of Sciences(中国科学院大学);Xiaohongshu Inc(小红书公司)
RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models
Beijing University of Posts and Telecommunications(北京邮电大学);The Hong Kong University of Science and Technology(香港科技大学);Beihang University(北京航空航天大学);University of the Chinese Academy of Sciences(中国科学院大学);ETH Zürich(苏黎世联邦理工学院);The Hong Kong Polytechnic University(香港理工大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Shanghai AI Lab(上海人工智能实验室)
AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models
Alibaba Group(阿里巴巴集团);School of Computer Science, Fudan University(复旦大学计算机科学技术学院);Shanghai Key Laboratory of Intelligent Information Processing, Fudan University(复旦大学上海市智能信息处理重点实验室)
Eliciting Causal Abilities in Large Language Models for Reasoning Tasks
BNU-HKBU United International College(北师港浸大);Hong Kong Baptist University(香港浸会大学);Beijing Normal University at Zhuhai(北京师范大学珠海校区);The Chinese University of Hong Kong(香港中文大学)
Reasoning Can Hurt the Inductive Abilities of Large Language Models
School of Information Sciences University of Illinois at Urbana-Champaign(信息科学学院伊利诺伊大学厄巴纳-香槟分校);Computer Science and Engineering HKUST(计算机科学与工程香港科技大学);Research Scientist, Intel Labs(英特尔实验室研究员);School of Information Sciences University of Illinois Urbana-Champaign(信息科学学院伊利诺伊大学厄巴纳-香槟分校)
KMIR: A Benchmark for Evaluating Knowledge Memorization, Identification and Reasoning Abilities of Language Models
Huawei Poisson Laboratory(华为泊松实验室);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Hong Kong University of Science and Technology(香港科技大学)
Atomic Thinking of LLMs: Decoupling and Exploring Mathematical Reasoning Abilities
Sun Yat-sen University(中山大学);Tsinghua University(清华大学);ByteDance Inc(字节跳动公司);Arizona State University(亚利桑那州立大学);Tencent Youtu Lab(腾讯优图实验室);Fudan University(复旦大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Guangdong Provincial Key Laboratory of Fire Science and Intelligent Emergency Technology(广东省消防科学与智能应急技术重点实验室);University of Illinois Chicago(伊利诺伊大学香槟分校)
daDPO: Distribution-Aware DPO for Distilling Conversational Abilities
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);University of Chicago(芝加哥大学);The Chinese University of Hong Kong(香港中文大学);East China Normal University(华东师范大学)
Code-driven Number Sequence Calculation: Enhancing the inductive Reasoning Abilities of Large Language Models
East China Normal University(东华大学);Fudan University(复旦大学);Shanghai Innovation Institute(上海创新研究院);Xi’an Jiaotong University(西安交通大学);Shanghai AI Laboratory(上海人工智能实验室)