In this work, we introduce a semiparametric token-sequence co-supervision training method. It trains a language model by simultaneously leveraging supervision from the traditional next token prediction loss which is calculated over the parametric token embedding space and the next sequence prediction loss which is calculated over the nonparametric sequence embedding space. The nonparametric sequence embedding space is constructed by a separate language model tasked to condense an input text into a single representative embedding. Our experiments demonstrate that a model trained via both supervisions consistently surpasses models trained via each supervision independently. Analysis suggests that this co-supervision encourages a broader generalization capability across the model. Especially, the robustness of parametric token space which is established during the pretraining step tends to effectively enhance the stability of nonparametric sequence embedding space, a new space established by another language model.
SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation
University of Science and Technology of China（中国科学技术大学）；Large Language Model Department, Tencent（腾讯大语言模型部门）；Shanghai Innovation Institute（上海创新研究院）；Zhongguancun Academy（中关村学院）
Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation
University of Chinese Academy of Sciences（中国科学院大学）；Chinese Information Processing Laboratory（中文信息处理实验室）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）
Not All Tokens Matter Equally: Dynamic In-context Vector Distillation with Decisive-Token Supervision for Long-form Medical Report Generation
UNSW Sydney（新南威尔士大学悉尼分校）；University of Technology Sydney（技术大学悉尼分校）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Stanford University（斯坦福大学）；Shanghai Jiao Tong University（上海交通大学）
Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancing
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Institute of Artificial Intelligence (TeleAI), China Telecom（人工智能研究院（TeleAI），中国电信）
Dense Is Not Enough: Hierarchical Supervision Allocation for Long-Horizon On-Policy Distillation
Ant Group（蚂蚁集团）；Alibaba International Digital Commerce Group（阿里巴巴国际数字商业集团）；University of Science and Technology of China（中国科学技术大学）；Peking University（北京大学）；University of Electronic Science and Technology of China（电子科技大学）
NITP: Next Implicit Token Prediction for LLM Pre-training
School of AI, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Xiaohongshu Inc.（小红书公司）；The Chinese University of Hong Kong（香港中文大学）；University of Science and Technology of China（中国科学技术大学）
ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models
Huazhong University of Science and Technology（华中科技大学）；Zhongguancun Academy（中关村学院）；DeepCybo；Harbin Institute of Technology（哈尔滨工业大学）；Beihang University（北京航空航天大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Zhengzhou University（郑州大学）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）
Escaping the KL Agreement Trap in On-Policy Distillation
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong Polytechnic University（香港理工大学）；Eastern Institute of Technology, Ningbo（宁波东方理工大学）
