Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models
LUMIA Lab（LUMIA实验室）；School of Artificial Intelligence（人工智能学院）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；Department of Electronic Engineering（电子工程系）；College of Artificial Intelligence（人工智能学院）；Tsinghua University（清华大学）；Shanghai Innovation Institute（上海创新研究院）
Yuliang Liu, Yunchong Song, Yixuan Wang, Kewen Ge, Alex Lamb, Qipeng Guo, Kai Chen, Bowen Zhou, Zhouhan Lin
We propose Next Concept Prediction (NCP), a generative pretraining paradigm built on top of Next Token Prediction (NTP). NCP predicts discrete concepts that span multiple tokens, thereby forming a more challenging pretraining objective. Our model, ConceptLM, quantizes hidden states using Vector Quantization and constructs a concept vocabulary. It leverages both NCP and NTP to drive parameter updates and generates a concept to guide the generation of the following tokens. We train ConceptLM from scratch at scales ranging from 70M to 1.5B parameters with up to 300B training data, including Pythia and GPT-2 backbones. Results on 13 benchmarks show that NCP yields consistent performance gains over traditional token-level models. Furthermore, continual pretraining experiments on an 8B-parameter Llama model indicate that NCP can further improve an NTP-trained model. Our analysis suggests that NCP leads to more powerful language models by introducing a harder pretraining task, providing a promising path toward better language modeling.
ImpRIF: Stronger Implicit Reasoning Leads to Better Complex Instruction Following
ByteDance China（字节跳动中国）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）
I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data?
Responsible AI Research Centre（负责任人工智能研究中心）；Australian Institute for Machine Learning（澳大利亚机器学习研究所）；Adelaide University（阿德莱德大学）；School of Computer Science and Engineering（计算机科学与工程学院）；The University of New South Wales（新南威尔士大学）；Halıcıoğlu Data Science Institute（哈利奇奥格卢数据科学研究所）；University of California San Diego（加州大学圣地亚哥分校）；School of Mathematics and Statistics（数学与统计学学院）；The University of Melbourne（墨尔本大学）
LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning
School of AI, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Kling Team, Kuaishou Technology（快手科技 Kling 团队）；Peking University（北京大学）；HKUST（香港科技大学）；CASIA（中国科学院自动化研究所）；Nanjing University（南京大学）；Renmin University of China（中国人民大学）；Tsinghua University（清华大学）
本文提出LatentOmni框架，通过统一的音频-视觉潜在空间进行多模态推理，利用特征级监督和Omni-Sync Position Embedding保持时间一致性，从而在多个音频-视觉推理基准测试中取得最佳性能。
LoopRPT: Reinforcement Pre-Training for Looped Language Models
Harbin Institute of Technology, Harbin, China（哈尔滨工业大学）；Tsinghua University, Beijing, China（清华大学）；The Hong Kong University of Science and Technology, Guangzhou, China（香港科学与技术大学）
"You are grounded!": Latent Name Artifacts in Pre-trained Language Models
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；University of Maryland, College Park（马里兰大学学院市分校）
预训练的语言模型（LM）可能会将源自其训练语料库的偏见延续到下游模型。我们专注于与给定名称的表示相关的工件（例如，Donald），其取决于语料库，可以与特定实体相关联，如由下一个令牌预测（例如，特朗普）。虽然在某些情况下是有帮助的，但接地也发生在不明确或不适当的情况下。例如，“Donald is a”的结尾与其他名字的结尾有很大不同，并且往往具有高于平均水平的负面情绪。我们展示了潜在的影响下游任务与阅读理解探针名称扰动改变模型的答案。作为一线希望，我们的实验表明，在不同语料库上进行额外的预训练可能会减轻这种偏见。
