Multi-Token Prediction (MTP) is now a common ingredient in language-model pre-training, as it improves downstream performance and enables faster inference through speculative decoding. We identify a flaw in standard MTP training, in which a residual component of the MTP gradient interferes with both next-token and multi-token learning, increasingly so with scale. Specifically, the MTP cross-entropy gradient can be decomposed exactly into a KL self-distillation term that pulls each MTP head toward the model's own next-token prediction (NTP) distribution and improves speculative decoding, and a residual that has a low signal-to-noise ratio and buries the useful KL signal. We introduce the KL gradient-recovery trick: when two distributions share a linear output head, the gradient of their KL divergence can be recovered exactly by combining their cross-entropy gradients through straight-through gradient routing. Unlike minimizing the KL directly, this trick requires no vocabulary-sized logit matrices and is compatible with standard fused cross-entropy kernels. This enables Soft Multi-Token Prediction (Soft-MTP), a one-line code change to MTP that removes the noisy residual and backpropagates only the KL self-distillation signal by reusing the existing NTP and MTP cross-entropy computations. Soft-MTP supports both DeepSeek- and Meta-style MTP, adds no parameters, and incurs no measurable overhead. From 167M to 1.2B parameters, Soft-MTP improves over standard MTP with lower NTP perplexity, faster speculative decoding and higher zero-shot accuracy at every scale, lower MTP perplexity from 689M onward, and stronger math and code reasoning after fine-tuning at 1.2B. Soft-MTP also matches, if not exceeds, direct KL divergence minimization on most metrics. Soft-MTP is, in effect, a free lunch for MTP optimization. Our code can be found at https://github.com/JaydenTeoh/Soft-MTP.
AutoEval Done Right: Using Synthetic Data for Model Evaluation
Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, USA（电子工程与计算机科学系，加州大学伯克利分校）；Department of Systems Immunology, Weizmann Institute of Science, Rehovot, Israel（系统免疫学系，魏茨曼科学研究所）；Inria, Ecole Normale Supérieure, Paris, France（法国国家信息与自动化技术研究所，巴黎高等师范学院）
Elastic Weight Consolidation Done Right for Continual Learning
School of Artificial Intelligence, Sun Yat-sen University, China（中山大学人工智能学院，中国）；Key Laboratory of Intelligent Assessment Technology for Sustainable Tourism, Ministry of Culture and Tourism, Sun Yat-sen University（文化旅游可持续发展智能评估技术重点实验室，文化部，中山大学）；Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China（机器智能与高级计算重点实验室，教育部，中国）
A Multi-layered Acoustic Tokenizing Deep Neural Network (MAT-DNN) for Unsupervised Discovery of Linguistic Units and Generation of High Quality Features
Transformer with Bidirectional Decoder for Speech Recognition
Tsinghua University（清华大学）；ShanghaiTech University（上海科技大学）；Tsingmicro Intelligent Technology Co., Limited（清微智能科技有限公司）；Beijing National Research Center For Information Science And Technology（北京国家信息科学与技术研究中心）
