Current techniques for post-training Large Language Models (LLMs) rely either on costly human supervision or on external verifiers to boost performance on tasks such as mathematical reasoning and code generation. However, as LLMs improve their problem-solving, any further improvement will potentially require high-quality solutions to difficult problems that are not available to humans. As a result, learning from unlabeled data is becoming increasingly attractive in the research community. Existing methods extract learning signal from a model's consistency, either by majority voting or by converting the model's internal confidence into reward. Although internal consistency metric such as entropy or self-certainty require no human intervention, as we show in this work, these are unreliable signals for large-scale and long-term training. To address the unreliability, we propose PRISM, a unified training framework that uses a Process Reward Model (PRM) to guide learning alongside model's internal confidence in the absence of ground-truth labels. We show that effectively combining PRM with self-certainty can lead to both stable training and better test-time performance, and also keep the model's internal confidence in check. Code available at https://github.com/ghimiremukesh/PRISM.
Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室);University of Chinese Academy of Sciences(中国科学院大学);Xiaohongshu Inc(小红书公司)
RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domains
University at Albany(阿尔巴尼大学);Rutgers University(罗格斯大学);Purdue University(普渡大学);Independent Researcher(独立研究员);Emory University(埃默里大学);Georgia Institute of Technology(佐治亚理工学院)
Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric
Qwen Large Model Application Team, Alibaba(阿里巴巴文勤大模型应用团队);Beijing University Of Posts and Telecommunications(北京邮电大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
BeamPERL: Parameter-Efficient RL with Verifiable Rewards Specializes Compact LLMs for Structured Beam Mechanics Reasoning
Department of Mechanical Engineering(机械工程系);Massachusetts Institute of Technology(麻省理工学院);Department of Civil and Environmental Engineering(土木与环境工程系);Schwarzman College of Computing(施瓦茨曼计算学院)
Instructing LLMs to Negotiate using Reinforcement Learning with Verifiable Rewards
Purdue University(普渡大学);California Institute of Technology(加州理工学院);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Massachusetts Institute of Technology(麻省理工学院)
Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China(中国科学院软件研究所信息处理实验室);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学);Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China(中国科学院大学网络安全学院);National Computer Network Emergency Response Technical Team/Coordination Center of China, Beijing, China(中国国家计算机网络应急技术配合中心)
RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution
National Key Laboratory for Novel Software Technology(新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);Huawei Technologies Co., Ltd.(华为技术有限公司)
Auditing Data Membership in Reinforcement Learning With Verifiable Rewards
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Shanghai Jiao Tong University(上海交通大学);Shandong University(山东大学);Xidian University(西安电子科技大学);Wuhan University(武汉大学)