Test-time inference has emerged as a powerful paradigm for enabling language models to ``think'' longer and more carefully about complex challenges, much like skilled human experts. While reinforcement learning (RL) can drive self-improvement in language models on verifiable tasks, some models exhibit substantial gains while others quickly plateau. For instance, we find that Qwen-2.5-3B far exceeds Llama-3.2-3B under identical RL training for the game of Countdown. This discrepancy raises a critical question: what intrinsic properties enable effective self-improvement? We introduce a framework to investigate this question by analyzing four key cognitive behaviors -- verification, backtracking, subgoal setting, and backward chaining -- that both expert human problem solvers and successful language models employ. Our study reveals that Qwen naturally exhibits these reasoning behaviors, whereas Llama initially lacks them. In systematic experimentation with controlled behavioral datasets, we find that priming Llama with examples containing these reasoning behaviors enables substantial improvements during RL, matching or exceeding Qwen's performance. Importantly, the presence of reasoning behaviors, rather than correctness of answers, proves to be the critical factor -- models primed with incorrect solutions containing proper reasoning patterns achieve comparable performance to those trained on correct solutions. Finally, leveraging continued pretraining with OpenWebMath data, filtered to amplify reasoning behaviors, enables the Llama model to match Qwen's self-improvement trajectory. Our findings establish a fundamental relationship between initial reasoning behaviors and the capacity for improvement, explaining why some language models effectively utilize additional computation while others plateau.
When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners
Harbin Institute of Technology(哈尔滨工业大学);Singapore Management University(新加坡管理学院);Monash University(墨尔本大学);Singapore University of Technology and Design(新加坡科技设计大学);National University of Singapore(国立新加坡大学)
Kwai-STaR: Transform LLMs into State-Transition Reasoners
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Kuaishou Technology(快手科技);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Pengcheng Laboratory(鹏城实验室)
Synergizing Large Language Models and Theory for Human-like Causal Reasoning
Center for Data Science, AAIS, Peking University(数据科学中心,AAIS,北京大学);Wangxuan Institute of Computer Technology, Peking University(王轩计算机技术研究所,北京大学);Tsinghua University(清华大学);State Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室)
Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond
School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院);Shaanxi Provincial Key Laboratory of Big Data Knowledge Engineering(陕西省大数据知识工程重点实验室);National Engineering Lab for Big Data Analytics(国家大数据分析工程实验室);School of Computer Science and Engineering, Nanyang Technological University(南洋理工大学计算机科学与工程学院)