The foundational capabilities of large language models are acquired during pretraining on internet-scale, highly heterogeneous data mixtures. In this work, we investigate an interesting geometric question regarding the converged state of pretraining: Does the model converge to a common minimizer across all data sources (e.g., Fig. 2b), or merely a minimizer of the summed loss (e.g., Fig. 2a)? We hypothesize that the geometric "closeness" of task-specific minima is intrinsically linked to downstream generalization. We reveal that standard optimizers (e.g., AdamW) often converge to points where task-specific minima are distant from each other. To address this, we propose the Nexus optimizer, which encourages the closeness of these minima by maximizing gradient similarity during optimization. Experiments across models ranging from 130M to 3B parameters, various data mixtures and hyperparameter schedules, show that Nexus significantly boosts downstream performance, despite achieving the same pretraining loss (see Fig. 1). Notably, on the 3B model, Nexus reduces the out-of-distribution loss by 0.012 and yields up to a 15.0% accuracy improvement on complex reasoning tasks (e.g., GSM8k). This finding challenges the reliance on pretraining loss as the sole proxy for model evaluation and demonstrates the importance of implicit biases in unlocking downstream generalization.
A Pretraining-Finetuning Computational Framework for Material Homogenization
Tsinghua University（清华大学）；Hainan Normal University（海南师范大学）；Bauhaus-Universität Weimar（魏玛包豪斯大学）；Leibniz University（莱布尼茨大学）；Queensland University of Technology（昆士兰科技大学）；Umeå University（于默奥大学）
From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers
The University of Tokyo（东京大学）；RIKEN AIP（理化学研究所AIP）；Flatiron Institute（Flatiron研究所）；CNRS（法国国家科学研究中心）；Université Côte d’Azur（蔚蓝海岸大学）；New York University（纽约大学）
Simulation-Based Pretraining and Domain Adaptation for Astronomical Time Series with Minimal Labeled Data
Massachusetts Institute of Technology, Cambridge, MA 02139, USA；Irvington High School, Fremont, CA 94538, USA；Center for Astrophysics, Harvard \& Smithsonian, Cambridge, MA 02138, USA
Towards Understanding Self-Pretraining for Sequence Classification
Unit of Artificial Intelligence & Computer Systems, Università Campus Bio-Medico di Roma（人工智能与计算机系统单位，罗马生物医学学院）；Unit of Advanced Robotics and Human-Centered Technologies, Università Campus Bio-Medico di Roma（先进机器人与以人为本技术单位，罗马生物医学学院）；Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University（诊断与介入部门，辐射物理，生物医学工程，乌梅拉大学）；Max Planck Institute for Intelligent Systems（智能系统马克斯·普朗克研究所）；ELLIS Institute Tübingen（图宾根ELLIS研究所）；Tübingen AI Center（图宾根人工智能中心）
Multitask Battery Management with Flexible Pretraining
IIIS, Tsinghua University（清华大学交叉信息研究院）；Shanghai Qi Zhi Institute（上海期智研究院）；State Key Laboratory of Intelligent Green Vehicle and Mobility, School of Vehicle and Mobility, Tsinghua University（清华大学智能绿色车辆与交通全国重点实验室、车辆与运载学院）；School of Advanced Manufacturing and Robotics, National Engineering Laboratory for Big Data Analysis and Applications, Peking University（北京大学先进制造与机器人学院、大数据分析与应用技术国家工程实验室）
Adaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve?
Concordia University（康考迪亚大学）；Mila-Quebec AI Institute（米拉-魁北克人工智能研究所）；University of Tübingen（蒂宾根大学）；Tübingen AI Center（蒂宾根人工智能中心）；Cohere For AI（Cohere人工智能项目）；Laval University（拉瓦尔大学）
Quebec Artificial Intelligence Institute (Mila)（魁北克人工智能研究所（米拉））；Université de Montréal（蒙特利尔大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of Athens（雅典大学）
