arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.06155cs.LGcs.AIcs.CL
收藏

迈向一致的世界模型:多令牌预测与潜在语义增强

Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancement

Shenzhen University(深圳大学);Microsoft Research Asia(微软亚洲研究院)

Qimin Zhong, Hao Liao, Haiming Qin, Mingyang Zhou, Rui Mao, Wei Chen, Naipeng Chao

AI总结:

本文通过多令牌预测和潜在语义增强方法,解决大语言模型内部世界模型一致性问题,提升表示对齐性和鲁棒性。

摘要:

是否大型语言模型(LLMs)能发展出连贯的内部世界模型仍是一个核心争论点。尽管传统下一步预测(NTP)专注于一步监督,多令牌预测(MTP)在学习更结构化的表示方面显示出潜力。在本文中,我们从理论角度分析了MTP的梯度归纳偏差,通过实证证据表明,MTP通过梯度耦合诱导表示收缩性,促进向内部信念状态的收敛。然而,我们发现标准MTP常出现结构幻觉,即离散令牌监督鼓励潜在空间中的非法捷径,违反环境约束。为解决此问题,我们提出了一种新的方法,即潜在语义增强MTP(LSE-MTP),该方法将预测锚定在真实隐藏状态轨迹上。在合成图和现实世界曼哈顿出租车行程实验中,LSE-MTP有效弥合了离散令牌与连续状态表示之间的差距,增强了表示对齐性,减少了结构幻觉,并提高了对扰动的鲁棒性。

英文摘要:

Whether Large Language Models (LLMs) develop coherent internal world models remains a core debate. While conventional Next-Token Prediction (NTP) focuses on one-step-ahead supervision, Multi-Token Prediction (MTP) has shown promise in learning more structured representations. In this work, we provide a theoretical perspective analyzing the gradient inductive bias of MTP, supported by empirical evidence, showing that MTP promotes the convergence toward internal belief states by inducing representational contractivity via gradient coupling. However, we reveal that standard MTP often suffers from structural hallucinations, where discrete token supervision encourages illegal shortcuts in latent space that violate environmental constraints. To address this, we propose a novel method Latent Semantic Enhancement MTP (LSE-MTP), which anchors predictions to ground-truth hidden state trajectories. Experiments on synthetic graphs and real-world Manhattan Taxi Ride show that LSE-MTP effectively bridges the gap between discrete tokens and continuous state representations, enhancing representation alignment, reducing structural hallucinations, and improving robustness to perturbations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑