Reinforcement learning with verifiable rewards (RLVR) has been shown to enhance the reasoning capabilities of large language models (LLMs), enabling the development of large reasoning models (LRMs). However, LRMs such as DeepSeek-R1 and OpenAI o1 often generate verbose responses containing redundant or irrelevant reasoning step-a phenomenon known as overthinking-which substantially increases computational costs. Prior efforts to mitigate this issue commonly incorporate length penalties into the reward function, but we find they frequently suffer from two critical issues: length collapse and training collapse, resulting in sub-optimal performance. To address them, we propose a pipeline for training a Conciseness Reward Model (CRM) that scores the conciseness of reasoning path. Additionally, we introduce a novel reward formulation named Conciseness Reward Function (CRF) with explicit dependency between the outcome reward and conciseness score, thereby fostering both more effective and more efficient reasoning. From a theoretical standpoint, we demonstrate the superiority of the new reward from the perspective of variance reduction and improved convergence properties. Besides, on the practical side, extensive experiments on five mathematical benchmark datasets demonstrate the method's effectiveness and token efficiency, which achieves an 8.1% accuracy improvement and a 19.9% reduction in response token length on Qwen2.5-7B. Furthermore, the method generalizes well to other LLMs including Llama and Mistral. The implementation code and datasets are publicly available for reproduction: https://anonymous.4open.science/r/CRM.
Latent Reward Steering: An Adaptive Inference-Time Framework that Implicitly Promotes Cognitive Behaviors in Reasoning LLMs
Rutgers University(罗格斯大学);South China Agricultural University(华南农业大学);Columbia University(哥伦比亚大学);Fenz.AI;QuantaAlpha;Adobe;Santa Clara University(圣克拉拉大学);City University of Hong Kong(香港城市大学)
Seeing with You: Perception-Reasoning Coevolution for Multimodal Reasoning
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Zhejiang University(浙江大学)
Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning
Key Laboratory of Network Data Science(网络数据科学实验室);Technology, Institute of Computing Technology, Chinese Academy of Sciences(技术,计算技术研究所,中国科学院);State Key Laboratory of AI Safety(人工智能安全国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学);University of California, Merced(加州大学梅尔塞德斯分校)
Enhancing Table Reasoning with Deterministic Table-State Rewards
University of California, Los Angeles(加州大学洛杉矶分校);McGill University(麦吉尔大学);Université de Montréal(蒙特利尔大学);University College London(伦敦大学学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));University of Manitoba(曼尼托巴大学)
PACR: Progressively Ascending Confidence Reward for LLM Reasoning
Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院);Microsoft Research Asia (MSRA)(微软亚洲研究院);University of Illinois at Urbana-Champaign (UIUC)(伊利诺伊大学厄巴纳-香槟分校)
Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signals
Tongji University(同济大学);Shanghai AI Laboratory(上海人工智能实验室);Nanyang Technological University(南洋理工大学);University of Science and Technology of China(中国科学技术大学);EPFL(苏黎世联邦理工学院);Wuhan University(武汉大学)
Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University(上海数据科学关键实验室,计算机科学学院,复旦大学)
InfoDensity: Rewarding Information-Dense Traces for Efficient Reasoning
Institute for Infocomm Research (I 2 R), A*STAR, Singapore(信息与通信研究机构(I 2 R),A*STAR,新加坡);Centre for Frontier AI Research (CFAR), A*STAR, Singapore(前沿人工智能研究中心(CFAR),A*STAR,新加坡)
Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models
the University of Hong Kong(香港大学);Tsinghua University(清华大学);Georgia Institute of Technology(佐治亚理工学院);National University of Singapore(新加坡国立大学);Zhejiang University(浙江大学);Renmin University of China(中国人民大学);the Chinese University of Hong Kong(香港中文大学)
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of California, San Diego(加州大学圣地亚哥分校);Texas A&M University(德克萨斯A&M大学);Stevens Institute of Technology(史蒂文斯理工学院)
FlowRL: Matching Reward Distributions for LLM Reasoning
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Microsoft Research(微软研究院);Tsinghua University(清华大学);Peking University(北京大学);Renmin University of China(中国人民大学);Stanford University(斯坦福大学);Toyota Technological Institute at Chicago(芝加哥丰田技术研究所)
BeamPERL: Parameter-Efficient RL with Verifiable Rewards Specializes Compact LLMs for Structured Beam Mechanics Reasoning
Department of Mechanical Engineering(机械工程系);Massachusetts Institute of Technology(麻省理工学院);Department of Civil and Environmental Engineering(土木与环境工程系);Schwarzman College of Computing(施瓦茨曼计算学院)
ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward
LMU Munich(慕尼黑大学);Harvard University(哈佛大学);University of Cambridge(剑桥大学);Mina AI;Konrad Zuse School of Excellence in Reliable AI (relAI)(康拉德·楚泽可靠人工智能卓越学校(relAI))