When writing and talking, people sometimes pause to think. Although reasoning-focused works have often framed reasoning as a method of answering questions or completing agentic tasks, reasoning is implicit in almost all written text. For example, this applies to the steps not stated between the lines of a proof or to the theory of mind underlying a conversation. In the Self-Taught Reasoner (STaR, Zelikman et al. 2022), useful thinking is learned by inferring rationales from few-shot examples in question-answering and learning from those that lead to a correct answer. This is a highly constrained setting -- ideally, a language model could instead learn to infer unstated rationales in arbitrary text. We present Quiet-STaR, a generalization of STaR in which LMs learn to generate rationales at each token to explain future text, improving their predictions. We address key challenges, including 1) the computational cost of generating continuations, 2) the fact that the LM does not initially know how to generate or use internal thoughts, and 3) the need to predict beyond individual next tokens. To resolve these, we propose a tokenwise parallel sampling algorithm, using learnable tokens indicating a thought's start and end, and an extended teacher-forcing technique. Encouragingly, generated rationales disproportionately help model difficult-to-predict tokens and improve the LM's ability to directly answer difficult questions. In particular, after continued pretraining of an LM on a corpus of internet text with Quiet-STaR, we find zero-shot improvements on GSM8K (5.9%$\rightarrow$10.9%) and CommonsenseQA (36.3%$\rightarrow$47.2%) and observe a perplexity improvement of difficult tokens in natural text. Crucially, these improvements require no fine-tuning on these tasks. Quiet-STaR marks a step towards LMs that can learn to reason in a more general and scalable way.
Can Large Models Teach Student Models to Solve Mathematical Problems Like Human Beings? A Reasoning Distillation Method via Multi-LoRA Interaction
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education（新一代人工智能技术及其交叉应用重点实验室（东南大学），教育部）
Teaching LLMs to Think Mathematically: A Critical Study of Decision-Making via Optimization
Data Science Department, Princess Sumaya University for Technology（普里姆斯萨大学技术学院数据科学系）；Electrical and Computer Engineering Department, Virginia Tech（弗吉尼亚理工大学电气与计算机工程系）；Computer Science Department, Princess Sumaya University for Technology（普里姆斯萨大学技术学院计算机科学系）；Computer Science Department, King Fahd University of Petroleum and Minerals（国王法赫德石油与矿物大学计算机科学系）
Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
University of Maryland（马里兰大学）；New York University（纽约大学）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；University of North Carolina（北卡罗来纳大学）；ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center（图宾根ELLIS研究所、马克斯·普朗克智能系统研究所、图宾根人工智能中心）；Columbia University（哥伦比亚大学）
Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoning
Zhejiang University（浙江大学）；Shanghai AI Laboratory（上海人工智能实验室）；Peking University（北京大学）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；University of Science and Technology of China（中国科学技术大学）；Shanghai Jiao Tong University（上海交通大学）
PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinking
Center for Computational Science and Engineering（计算科学与工程中心）；Schmarzman College of Computing（施马茨曼计算学院）；Laboratory for Atomistic and Molecular Mechanics (LAMM)（原子与分子力学实验室）；Massachusetts Institute of Technology（麻省理工学院）
