Constraint-based Pre-training: From Structured Constraints to Scalable Model Initialization
School of Computer Science and Engineering, Southeast University, Nanjing, China（东南大学计算机科学与工程学院，南京，中国）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及其跨学科应用重点实验室（东南大学），教育部，中国）
Fu Feng, Yucheng Xie, Ruixiao Shi, Jing Wang, Xin Geng
The pre-training and fine-tuning paradigm has become the dominant approach for model adaptation. However, conventional pre-training typically yields models at a fixed scale, whereas practical deployment often requires models of varying sizes, exposing its limitations when target model scales differ from those used during pre-training. To address this, we propose an innovative constraint-based pre-training paradigm that imposes structured constraints during pre-training to disentangle size-agnostic knowledge into reusable weight templates, while assigning size-specific adaptation to lightweight weight scalers, thereby reformulating variable-sized model initialization as a multi-task adaptation problem. Within this paradigm, we further introduce WeiT, which employs Kronecker-based constraints to regularize the pre-training process. Specifically, model parameters are represented as compositions of weight templates via concatenation and weighted aggregation, with adaptive connections governed by lightweight weight scalers whose parameters are learned from limited data. This design enables flexible and efficient construction of model weights across diverse downstream scales. Extensive experiments demonstrate the efficiency and effectiveness of WeiT, achieving state-of-the-art performance in initializing models with varying depths and widths across a broad range of perception and embodied learning tasks, including Image Classification, Image Generation, and Embodied Control. Moreover, its effectiveness generalizes to both Transformer-based and Convolution-based architectures, consistently enabling faster convergence and improved performance even under full training.
Self-Supervised Weight Templates for Scalable Vision Model Initialization
School of Computer Science and Engineering, Southeast University, Nanjing, China（计算机科学与工程学院，东南大学，南京，中国）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及其交叉应用重点实验室（东南大学），教育部，中国）
Scalable Complexity Control Facilitates Reasoning Ability of LLMs
Institute of Natural Sciences, School of Mathematical Sciences, Shanghai Jiao Tong University（上海交通大学数学科学学院自然科学院）；MOE-LSC, School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Center for Machine Learning Research, School of Mathematical Sciences, Peking University（北京大学数学科学学院机器学习研究中心）；Center for LLM, Institute for Advanced Algorithms Research, Shanghai（上海高级算法研究所大语言模型中心）；MemTensor (Shanghai) Technology Co., Ltd.（MemTensor（上海）科技有限公司）
Disentangling Structured Components: Towards Adaptive, Interpretable and Scalable Time Series Forecasting
Australian Artificial Intelligence Institute, University of Technology Sydney（悉尼科技大学澳大利亚人工智能研究所）；Southern University of Science and Technology（南方科技大学）；University of California, Los Angeles（加州大学洛杉矶分校）；University of Tokyo（东京大学）；University of New South Wales（新南威尔士大学）；Tencent WXG（腾讯微信事业群）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）
Structured Graph Learning for Scalable Subspace Clustering: From Single-view to Multi-view
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；School of Resources and Environment, University of Electronic Science and Technology of China（电子科技大学资源与环境学院）
Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
John Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·保罗森工程与应用科学学校）；Center for Brain Sciences（脑科学研究中心）；Kempner Institute（凯普纳研究所）
Weights initialization of neural networks for function approximation
School of Mathematics and Computational Science（数学与计算科学学院）；National Center for Applied Mathematics in Hunan（湖南应用数学研究中心）；Hunan Key Laboratory for Computation and Simulation in Science and Engineering（计算与工程科学与工程计算模拟湖南重点实验室）；University of Texas at El Paso（德克萨斯埃尔帕索大学）
DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；University of International Business and Economics, China（对外经济贸易大学）；XiaoMi AI Lab, China（小米AI实验室）
One-Cycle Structured Pruning via Stability-Driven Subnetwork Search
IT Application Research Center, Korea Electronics Technology Institute（韩国电子技术研究所应用信息技术中心）；Department of Intelligent Semiconductors, Soongsil University（顺慈大学智能半导体系）；Department of Computer Science & Engineering, Seoul National University（首尔国立大学计算机科学与工程系）
A Scalable Factorization Approach for High-Order Structured Tensor Recovery
Department of Computer Science and Engineering, the Ohio State University（计算机科学与工程系，俄亥俄州立大学）；Department of Electrical Engineering, Colorado School of Mines（电气工程系，科罗拉多矿学院）
