Despite the remarkable reasoning performance, eliciting the long chain-of-thought (CoT) ability in large language models (LLMs) typically requires costly reinforcement learning or supervised fine-tuning on high-quality distilled data. We investigate the internal mechanisms behind this capability and show that a small set of high-impact activations in the last few layers largely governs long-form reasoning attributes, such as output length and self-reflection. By simply amplifying these activations and inserting "wait" tokens, we can invoke the long CoT ability without any training, resulting in significantly increased self-reflection rates and accuracy. Moreover, we find that the activation dynamics follow predictable trajectories, with a sharp rise after special tokens and a subsequent exponential decay. Building on these insights, we introduce a general training-free activation control technique. It leverages a few contrastive examples to identify key activations, and employs simple analytic functions to modulate their values at inference time to elicit long CoTs. Extensive experiments confirm the effectiveness of our method in efficiently eliciting long CoT reasoning in LLMs and improving their performance. Additionally, we propose a parameter-efficient fine-tuning method that trains only a last-layer activation amplification module and a few LoRA layers, outperforming full LoRA fine-tuning on reasoning benchmarks with significantly fewer parameters. Our code and data are publicly released.
Shanghai Jiao Tong University（上海交通大学）；Microsoft Research Asia（微软亚洲研究院）；Sea AI Lab（Sea人工智能实验室）
Numerical reasoning over natural language has been a long-standing goal for the research community. However, cutting-edge language models have proven difficult to reliably generalize to a broad range of numbers, although they have shown proficiency in reasoning over common and simple numbers. In this paper, we propose a novel method to elicit and exploit the numerical reasoning knowledge hidden in pre-trained language models using simple anchor numbers. Concretely, we first leverage simple numbers as anchors to probe the implicitly inferred arithmetic expressions from language models, and then…
On the Thinking-Language Modeling Gap in Large Language Models
Hong Kong Baptist University（香港 Baptist 大学）；MBZUAI Carnegie Mellon University（MBZUAI 美国卡内基梅隆大学）；The University of Sydney（悉尼大学）；The Chinese University of Hong Kong（香港中文大学）
Re-Reading Improves Reasoning in Large Language Models
Institute of Information Engineering, CAS（中国科学院信息工程研究所）；SKLSDE Lab, Beihang University（北京航空航天大学）；University of Technology Sydney（悉尼科技大学）；Microsoft Corporation（微软公司）
Eliciting Medical Reasoning with Knowledge-enhanced Data Synthesis: A Semi-Supervised Reinforcement Learning Approach
College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与技术学院）；Shanghai AI Laboratory（上海人工智能实验室）；CMIC, Shanghai Jiao Tong University（上海交通大学CMIC）；School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Department of Radiology, Shanghai Sixth People’s Hospital Affiliated to Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属第六人民医院放射科）；Institute of Artificial Intelligence for Medicine, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院人工智能医学研究所）
Abstraction-of-Thought Makes Language Models Better Reasoners
Institute for Artificial Intelligence, Tsinghua University（清华大学人工智能学院）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tencent AI Lab（腾讯AI实验室）
Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMs
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Southeast University（东南大学）
MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs
UC Santa Cruz（加州大学圣克鲁兹分校）；University of British Columbia（不列颠哥伦比亚大学）；Nanyang Technological University（南洋理工大学）；Stanford University（斯坦福大学）；Weill Cornell Medicine（威尔康奈尔医学院）；NYU Langone Health（纽约大学兰贡医学中心）；Chungnam National University Sejong Hospital（忠南大学世宗医院）；Vector Institute（矢量研究院）；Pusan National University Hospital（釜山国立大学医院）；Massachusetts General Hospital（麻省总医院）
Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do
The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所复杂系统认知与决策智能实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Demystifying Entropy-based Selection for Chain-of-Thought Compression in Large Reasoning Models
University of Trieste（的里雅斯特大学）；University of Milano-Bicocca（米兰-比科卡大学）；University of Groningen（格罗宁根大学）；Khoury College of Computer Sciences, Northeastern University（东北大学科里计算机学院）
Beyond the Commitment Boundary: Probing Epiphenomenal Chain-of-Thought in Large Reasoning Models
CLCG, University of Groningen（格罗宁根大学CLCG）；University of Milano-Bicocca（米兰-布雷拉大学）；University of Trieste（特里耶大学）；Khoury College of Computer Sciences, Northeastern University（东北大学Khoury计算机科学学院）
