发表机构
City University of Hong Kong; Huawei Technologies; Mohamed bin Zayed University of Artificial Intelligence; McGill University; Peking University（香港城市大学; 华为技术有限公司; 穆罕默德·本·扎耶德人工智能大学; 麦吉尔大学; 北京大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
AdaMTP是适配序列内在可预测性的自适应MTP训练范式，通过熵算法划分语义边界抑制噪声梯度，在三类主干模型的多任务基准中均优于标准MTP，兼具性能与推理速度优势。
AI 中文摘要
多令牌预测（Multi-Token Prediction, MTP）已成为一种有效范式，它在共享的大语言模型主干上增加辅助头，训练模型并行预测多个未来令牌，以丰富监督信号并加速推理。然而，现有训练框架采用固定长度的预测范围，忽略了自然语言和代码的信息密度高度非均匀的特性。强制辅助头跨越高熵语义边界进行预测，会注入嘈杂、冲突的训练信号；由于这些头共享主干的潜在表示，产生的梯度反向传播会干扰模型的核心能力。我们提出AdaMTP，一种自适应训练范式，可动态调整预测范围以匹配序列的内在可预测性。其核心是一种基于熵的分段算法，利用基础模型检测不确定性的突然激增作为语义边界，将序列划分为可变长度的组。每个令牌被分配一个自适应预测深度，动态掩码的MTP目标会抑制跨越这些边界的预测损失，从而减弱损害主干的嘈杂梯度。在数学推理、代码生成以及三个主干（Llama-3.1-8B、Qwen-2.5-7B、Gemma-3-12B）上的通用基准测试中，AdaMTP在任务性能和推理加速方面均始终优于标准MTP。
英文摘要
Multi-Token Prediction (MTP) has emerged as an effective paradigm that augments a shared Large Language Model backbone with auxiliary heads, training the model to predict several future tokens in parallel to enrich its supervision signal and accelerate inference. However, existing training frameworks adopt a rigid, fixed-length prediction horizon, disregarding the highly non-uniform information density of natural language and code. Forcing the auxiliary heads to predict across high-entropy semantic boundaries injects noisy, conflicting training signals; because these heads share the backbone's latent representations, the resulting gradients backpropagate and interfere with the model's core capabilities. We propose AdaMTP, an adaptive training paradigm that dynamically aligns the prediction horizon with the intrinsic predictability of the sequence. At its core, an entropy-based segmentation algorithm leverages the base model to detect sudden surges in uncertainty as semantic boundaries, partitioning sequences into variable-length groups. Each token is assigned an adaptive prediction depth, and a dynamically masked MTP objective suppresses the loss for predictions that cross these boundaries, attenuating the noisy gradients that degrade the backbone. Across mathematical reasoning, code generation, and general benchmarks on three backbones (Llama-3.1-8B, Qwen-2.5-7B, Gemma-3-12B), AdaMTP consistently outperforms standard MTP in both task performance and inference speedup.
