Latent reasoning via continuous chain-of-thoughts (Latent CoT) has emerged as a promising alternative to discrete CoT reasoning. Operating in continuous space increases expressivity and has been hypothesized to enable superposition: the ability to maintain multiple candidate solutions simultaneously within a single representation. Despite theoretical arguments, it remains unclear whether language models actually leverage superposition when reasoning using latent CoTs. We investigate this question across three regimes: a training-free regime that constructs latent thoughts as convex combinations of token embeddings, a fine-tuned regime where a base model is adapted to produce latent thoughts, and a from-scratch regime where a model is trained entirely with latent thoughts to solve a given task. Using Logit Lens and entity-level probing to analyze internal representations, we find that only models trained from scratch exhibit signs of using superposition. In the training-free and fine-tuned regimes, we find that the superposition either collapses or is not used at all, with models discovering shortcut solutions instead. We argue that this is due to two complementary phenomena: i) pretraining on natural language data biases models to commit to a token in the last layers ii) capacity has a huge effect on which solutions a model favors. Together, our results offer a unified explanation for when and why superposition arises in continuous chain-of-thought reasoning, and identify the conditions under which it collapses.
Do Large Language Models Show Biases in Causal Learning?
UBA(布宜诺斯艾利斯大学);Università degli Studi di Genova(热那亚大学);University of Buenos Aires(布宜诺斯艾利斯大学);Universidad Nacional del Sur(南方国立大学);IIIA-CSIC(西班牙国家研究委员会人工智能研究所);Universidad Autónoma de Barcelona(巴塞罗那自治大学);Arizona State University(亚利桑那州立大学)
Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning
The Hong Kong University of Science and Technology(香港科学与技术大学);University of Electronic Science and Technology of China(电子科技大学);Dalian University of Technology(大连理工大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);Zhejiang University(浙江大学);University of Oxford(牛津大学);NVIDIA(NVIDIA公司);University of Surrey(萨里大学)
Shanghai AI Laboratory(上海人工智能实验室);Harbin Institute of Technology(哈尔滨工业大学);Fudan University(复旦大学);CUHK(香港中文大学);Tsinghua University(清华大学);CUHK-Shenzhen(香港中文大学深圳校区);UESTC(电子科技大学);USTC(中国科学技术大学)
The Evolution of Thought: Tracking LLM Overthinking via Reasoning Dynamics Analysis
State Key Laboratory of AI Safety,Institute of Computing Technology, Chinese Academy of Sciences(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学)
Evaluating and Inducing Personality in Pre-trained Language Models
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Yuanpei College, Peking University(北京大学元培学院);National Key Laboratory of General Artificial Intelligence, BIGAI(北京通用人工智能研究院通用人工智能全国重点实验室);Beijing Jiaotong University(北京交通大学)