Whether Large Language Models (LLMs) develop coherent internal world models remains a core debate. While conventional Next-Token Prediction (NTP) focuses on one-step-ahead supervision, Multi-Token Prediction (MTP) has shown promise in learning more structured representations. In this work, we provide a theoretical perspective analyzing the gradient inductive bias of MTP, supported by empirical evidence, showing that MTP promotes the convergence toward internal belief states by inducing representational contractivity via gradient coupling. However, we reveal that standard MTP often suffers from structural hallucinations, where discrete token supervision encourages illegal shortcuts in latent space that violate environmental constraints. To address this, we propose a novel method Latent Semantic Enhancement MTP (LSE-MTP), which anchors predictions to ground-truth hidden state trajectories. Experiments on synthetic graphs and real-world Manhattan Taxi Ride show that LSE-MTP effectively bridges the gap between discrete tokens and continuous state representations, enhancing representation alignment, reducing structural hallucinations, and improving robustness to perturbations.
Instituto Superior Técnico, Universidade de Lisboa(里斯本大学理工学院);Instituto de Telecomunicações(电信研究所);TransPerfect(TransPerfect公司);Carnegie Mellon University(卡内基梅隆大学)
Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学首都人工智能学院);AI Platform, Xiaohongshu Inc.(小红书人工智能平台);University of Electronic Science and Technology of China(电子科技大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
SeLaR: Selective Latent Reasoning in Large Language Models
Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology(广东省超高清沉浸式媒体技术重点实验室);Shenzhen Graduate School, Peking University(北京大学深圳研究生院)
Token-Disentangled Latent Test-Time Scaling for Vision-Language Reasoning
Qwen Business Unit of Alibaba(阿里巴巴通义千问事业部);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术全国重点实验室);Zhejiang University(浙江大学);University of Waterloo(滑铁卢大学);Chinese Academy of Sciences(中国科学院);The Hong Kong University of Science and Technology(香港科技大学);Frontier Robotics(前沿机器人)
Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos
The University of Hong Kong(香港大学);ARC Lab, Tencent PCG(腾讯PCG实验室);The Chinese University of Hong Kong(香港中文大学);University of California, Berkeley(加州大学伯克利分校)
Image Tokens Matter: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing
Hasso Plattner Institute(霍普夫纳研究所);University of Potsdam(波茨坦大学);University of Cambridge(剑桥大学);University of Oxford(牛津大学);German University of Digital Science(德国数字科学大学)
LaTER: Efficient Test-Time Reasoning via Latent Exploration and Explicit Verification
University of Science and Technology of China(科学技术大学);Unisound AI Technology Co., Ltd(Unisound AI技术有限公司);MAIS, Institute of Automation, Chinese Academy of Sciences(自动化研究所,中国科学院)
SPOT: Span-level Pause-of-Thought for Efficient and Interpretable Latent Reasoning in Large Language Models
School of New Media and Communication, Tianjin University(新媒体与传播学院,天津大学);School of Computer Science and Engineering, Beihang University(计算机科学与工程学院,北航大学)
Byte Latent Transformer: Patches Scale Better Than Tokens
FAIR at Meta(Meta FAIR研究院);Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院);University of Chicago(芝加哥大学)
UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMs
University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学);National University of Singapore(新加坡国立大学);Zhongguancun Academy(中关村学院)
SimLens for Early Exit in Large Language Models: Eliciting Accurate Latent Predictions with One More Token
Institute of Neuroscience, State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences(中国科学院脑科学与智能技术卓越创新中心神经科学研究所,脑认知与脑启发智能技术国家重点实验室,脑科学与智能技术卓越创新中心);School of Future Technology, University of Chinese Academy of Sciences(中国科学院大学未来技术学院)
Inria(法国国家信息与自动化技术研究院);PSL Research University(巴黎社会科学高等研究院);CMAP(巴黎高等理工学院应用数学与计算科学实验室);Ecole Polytechnique Palaiseau, France(法国巴黎高等理工学院Palaiseau分校);Sakana AI Tokyo, Japan(日本东京Sakana AI公司)