Discrete diffusion models have recently become competitive with autoregressive models for language modeling, even outperforming them on reasoning tasks requiring planning and global coherence, but they require more computation at inference time. We trace this trade-off to a key mechanism: diffusion models are trained to jointly predict a distribution over all unknown tokens, including those that will not actually be decoded in the current step. Ablating this joint prediction yields faster inference but degrades performance, revealing that accurate prediction at the decoded position relies on joint reasoning about the distribution of undecoded tokens. We interpret these as latent tokens and introduce a method for modulating their number, demonstrating empirically that this enables a smooth tradeoff between inference speed and sample quality. Furthermore, we demonstrate that latent tokens can be introduced into autoregressive models through an auxiliary multi-token prediction objective, yielding substantial improvements on the same reasoning tasks where they have traditionally struggled. Our results suggest that latent tokens, while arising naturally in diffusion, represent a general mechanism for improving performance on tasks requiring global coherence or lookahead.
Inria(法国国家信息与自动化技术研究院);PSL Research University(巴黎社会科学高等研究院);CMAP(巴黎高等理工学院应用数学与计算科学实验室);Ecole Polytechnique Palaiseau, France(法国巴黎高等理工学院Palaiseau分校);Sakana AI Tokyo, Japan(日本东京Sakana AI公司)
ADaPT: Token-Level Decoupling for Efficient Large Reasoning Models
School of Data Science, Fudan University(复旦大学数据科学学院);Shanghai Institute of Artificial Intelligence for Education, East China Normal University(华东师范大学上海智能教育研究院);College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Ant Group(蚂蚁集团)
Towards Long-Horizon Interpretability: Efficient and Faithful Multi-Token Attribution for Reasoning LLMs
Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China(香港城市大学计算机科学系);Harbin Institute of Technology, Harbin, China(哈尔滨工业大学)
Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning
Tsinghua University(清华大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Peking University(北京大学);Qwen Large Model Application Team, Alibaba(阿里云文大模型应用团队);Microsoft(微软);Shenzhen Key Laboratory of Ubiquitous Data Enabling, Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院 ubiquitous 数据赋能重点实验室)
Beyond Dense States: Sparse Transcoders as Causally Testable Operators for LLM Latent Reasoning
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics(南京航空航天大学计算机科学与技术学院);MIIT Key Laboratory of Pattern Analysis and Machine Intelligence(信息产业部模式分析与机器智能重点实验室);Institute for AI, Tsinghua University, Beijing, China(清华大学人工智能研究院)
Latent-GRPO: Group Relative Policy Optimization for Latent Reasoning
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院人工智能安全国家重点实验室,计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Instituto Superior Técnico, Universidade de Lisboa(里斯本大学理工学院);Instituto de Telecomunicações(电信研究所);TransPerfect(TransPerfect公司);Carnegie Mellon University(卡内基梅隆大学)
TokenSqueeze: Performance-Preserving Compression for Reasoning LLMs
State Key Lab of CAD&CG, Zhejiang University(浙江大学CAD&CG国家重点实验室);School of Software Technology, Zhejiang University(浙江大学软件学院);Alibaba Cloud(阿里云);Zhiyuan Research Institute(智源研究院)
GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
School of Mathematical Sciences, Shanghai Jiao Tong University(上海交通大学数学科学学院);Institute of Natural Sciences, Shanghai Jiao Tong University(上海交通大学自然科学研究院)