RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution
National Key Laboratory for Novel Software Technology(新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);Huawei Technologies Co., Ltd.(华为技术有限公司)
Reinforcement learning from verifiable rewards (RLVR) stimulates the thinking processes of large language models (LLMs), substantially enhancing their reasoning abilities on verifiable tasks. It is often assumed that similar gains should transfer to general question answering (GQA), but this assumption has not been thoroughly validated. To assess whether RLVR automatically improves LLM performance on GQA, we propose a Cross-Generation evaluation framework that measures the quality of intermediate reasoning by feeding the generated thinking context into LLMs of varying capabilities. Our evaluation leads to a discouraging finding: the efficacy of the thinking process on GQA tasks is markedly lower than on verifiable tasks, suggesting that explicit training on GQA remains necessary in addition to training on verifiable tasks. We further observe that direct RL training on GQA is less effective than RLVR. Our hypothesis is that, whereas verifiable tasks demand robust logical chains to obtain high rewards, GQA tasks often admit shortcuts to high rewards without cultivating high-quality thinking. To avoid possible shortcuts, we introduce a simple method, Separated Thinking And Response Training (START), which first trains only the thinking process, using rewards defined on the final answer. We show that START improves both the quality of thinking and the final answer across several GQA benchmarks and RL algorithms.
CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR
Qwen Large Model Application Team, Alibaba Institute of Automation, Chinese Academy of Sciences(文勤大模型应用团队,阿里巴巴自动化研究所,中国科学院);Qwen Large Model Application Team, Alibaba(文勤大模型应用团队,阿里巴巴)
Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China(天津大学计算机科学与技术学院 TJUNLP 实验室,中国);Independent Researcher(独立研究者);College of Computer and Information Engineering, Tianjin Normal University, China(天津师范大学计算机与信息工程学院,中国)
TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs
University of Chinese Academy of Sciences(中国科学院大学);Meituan(美团);MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所模式识别国家重点实验室与智能信息处理重点实验室)
Linear Dynamics in the RLVR Training of Large Language Models
Department of Data Science, City University of Hong Kong(香港城市大学数据科学系);Hong Kong Institute of AI for Science, City University of Hong Kong(香港城市大学人工智能科学研究院);Li Auto Inc.;Beihang University(北航大学)
Rewarding How Models Think Pedagogically: Integrating Pedagogical Reasoning and Thinking Rewards for LLMs in Education
Chosun University(昌原大学);Korea University(韩国大学);Seoul National University(首尔国立大学);Korea Institute for Curriculum and Evaluation(韩国课程与评价研究院);Upstage;Indiana University Bloomington(印第安纳大学布卢明顿分校);Nanyang Technological University(南洋理工大学)
Generalization of RLVR Using Causal Reasoning as a Testbed
Johns Hopkins University(约翰霍普金斯大学);University of Maryland, College Park(马里兰大学学院公园分校);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Microsoft Research Asia(微软亚洲研究院);Toyota Technological Institute at Chicago(芝加哥丰田技术研究所)
Boosting LLM Exploration via Weak-Model Guidance in RLVR
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);National Engineering Research Center of New Electronic Publishing Technologies(国家新型电子出版技术工程研究中心)