Fine-tuning large language models (LLMs) on reasoning benchmarks via reinforcement learning requires a specific reward function, often binary, for each benchmark. This comes with two potential limitations: the need to design the reward, and the potentially sparse nature of binary rewards. Here, we systematically investigate rewards derived from the probability or log-probability of emitting the reference answer (or any other prompt continuation present in the data), which have the advantage of not relying on specific verifiers and being available at scale. Several recent works have advocated for the use of similar rewards (e.g., VeriFree, JEPO, RLPR, NOVER). We systematically compare variants of likelihood-based rewards with standard baselines, testing performance both on standard mathematical reasoning benchmarks, and on long-form answers where no external verifier is available. We find that using the log-probability of the reference answer as the reward for chain-of-thought (CoT) learning is the only option that performs well in all setups. This reward is also consistent with the next-token log-likelihood loss used during pretraining. In verifiable settings, log-probability rewards bring comparable or better success rates than reinforcing with standard binary rewards, and yield much better perplexity. In non-verifiable settings, they perform on par with SFT. On the other hand, methods based on probability, such as VeriFree, flatline on non-verifiable settings due to vanishing probabilities of getting the correct answer. Overall, this establishes log-probability rewards as a viable method for CoT fine-tuning, bridging the short, verifiable and long, non-verifiable answer settings.
Rewarding How Models Think Pedagogically: Integrating Pedagogical Reasoning and Thinking Rewards for LLMs in Education
Chosun University（昌原大学）；Korea University（韩国大学）；Seoul National University（首尔国立大学）；Korea Institute for Curriculum and Evaluation（韩国课程与评价研究院）；Upstage；Indiana University Bloomington（印第安纳大学布卢明顿分校）；Nanyang Technological University（南洋理工大学）
Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihood
Baidu Inc（百度公司）；College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；Key Laboratory of Symbolic Computation and Knowledge Engineering of MOE, Jilin University（吉林大学教育部符号计算与知识工程重点实验室）；State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院人工智能安全国家重点实验室）
Aligning Language Models Using Follow-up Likelihood as Reward Signal
National University of Singapore（新加坡国立大学）；Peking University（北京大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Tencent AI Lab（腾讯AI实验室）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；University of Science and Technology of China（中国科学技术大学）
Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学深圳研究院）；Institute of Artificial Intelligence (TeleAI), China Telecom Corp Ltd（中国电信人工智能研究院）；College of Integrated Circuits, Zhejiang University, Hangzhou, Zhejiang, China（浙江大学集成电路学院）；Zhongguancun Academy, Beijing, China（中关村学院）
Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood
College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；Key Laboratory of Symbolic Computation and Knowledge Engineering of MOE, Jilin University（教育部符号计算与知识工程重点实验室，吉林大学）；Baidu Inc.（百度公司）；Tsinghua University（清华大学）；State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院人工智能安全重点实验室，计算技术研究所）
ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding
Kuaishou Technology Co., Ltd.（快手科技有限公司）；Institute for Clarity in Documentation（文献清晰度研究所）；Inria Paris-Rocquencourt（法国国家信息与自动化研究所巴黎-罗康分所）；Rajiv Gandhi University（拉吉夫甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕尔默研究实验室）
R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Beijing Normal University（北京师范大学）
Rewarding Novel Deductions: Solver-guided Process Supervision for Logical Reasoning
Information Technology University（信息技术大学）；Huazhong Agricultural University（华中农业大学）；Qatar Computing Research Institute, Hamad Bin Khalifa University（卡塔尔计算研究所，哈马德·本·哈利法大学）
GRAM-R$^2$: Self-Training Generative Foundation Reward Models for Reward Reasoning
School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；Pattern Recognition Center, WeChat AI, Tencent Inc.（腾讯公司微信AI模式识别中心）
Disentangling Perception and Reasoning in Multimodal LLMs via Reward Design
Department of Computer Science, Dartmouth College（达特茅斯大学计算机科学系）；Department of Computer Science, University of Central Florida（中央佛罗里达大学计算机科学系）；Independent Researcher（独立研究者）
PACR: Progressively Ascending Confidence Reward for LLM Reasoning
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；Microsoft Research Asia (MSRA)（微软亚洲研究院）；University of Illinois at Urbana-Champaign (UIUC)（伊利诺伊大学厄巴纳-香槟分校）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California, San Diego（加州大学圣地亚哥分校）；Texas A&M University（德克萨斯A&M大学）；Stevens Institute of Technology（史蒂文斯理工学院）
Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signals
Tongji University（同济大学）；Shanghai AI Laboratory（上海人工智能实验室）；Nanyang Technological University（南洋理工大学）；University of Science and Technology of China（中国科学技术大学）；EPFL（苏黎世联邦理工学院）；Wuhan University（武汉大学）
InfoDensity: Rewarding Information-Dense Traces for Efficient Reasoning
Institute for Infocomm Research (I 2 R), A*STAR, Singapore（信息与通信研究机构（I 2 R），A*STAR，新加坡）；Centre for Frontier AI Research (CFAR), A*STAR, Singapore（前沿人工智能研究中心（CFAR），A*STAR，新加坡）
Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models
the University of Hong Kong（香港大学）；Tsinghua University（清华大学）；Georgia Institute of Technology（佐治亚理工学院）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）；Renmin University of China（中国人民大学）；the Chinese University of Hong Kong（香港中文大学）
FlowRL: Matching Reward Distributions for LLM Reasoning
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；Microsoft Research（微软研究院）；Tsinghua University（清华大学）；Peking University（北京大学）；Renmin University of China（中国人民大学）；Stanford University（斯坦福大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
BeamPERL: Parameter-Efficient RL with Verifiable Rewards Specializes Compact LLMs for Structured Beam Mechanics Reasoning
Department of Mechanical Engineering（机械工程系）；Massachusetts Institute of Technology（麻省理工学院）；Department of Civil and Environmental Engineering（土木与环境工程系）；Schwarzman College of Computing（施瓦茨曼计算学院）
Posterior-GRPO: Rewarding Reasoning Processes in Code Generation
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；The State Key Laboratory of Blockchain and Data Security, Zhejiang University（浙江大学区块链与数据安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）
ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward
LMU Munich（慕尼黑大学）；Harvard University（哈佛大学）；University of Cambridge（剑桥大学）；Mina AI；Konrad Zuse School of Excellence in Reliable AI (relAI)（康拉德·楚泽可靠人工智能卓越学校（relAI））
