收集大量高质量数据可能代价高昂或不切实际，并且是机器学习中的一个瓶颈。一种替代做法是，用来自更易获取来源的数据来扩充来自目标分布的一小组 n 个数据点，例如在不同情况下收集的数据，或由生成模型合成的数据。我们将此类数据称为“替代数据”。我们研究了一种用于将替代数据整合到训练中的加权经验风险最小化（ERM）方法。我们在若干经典统计模型下对该方法进行了数学分析，并在来自不同领域的数据集上通过实验验证了我们的发现。我们的主要发现是：（i）整合替代数据可以显著降低原始分布上的测试误差。令人惊讶的是，即使替代数据与原始数据无关，这种情况也可能发生。我们将这一行为追溯到经典的 Stein 悖论。（ii）为了获得替代数据的益处，使用最优加权 ERM 至关重要。（iii）在真实数据与替代数据的混合数据上训练的模型，其测试误差可近似由一条缩放定律描述。该缩放定律可用于预测最优加权方案，并选择要添加的替代数据数量。
英文摘要：
Collecting large quantities of high-quality data can be prohibitively expensive or impractical, and a bottleneck in machine learning. One may instead augment a small set of $n$ data points from the target distribution with data from more accessible sources, e.g. data collected under different circumstances or synthesized by generative models. We refer to such data as `surrogate data'. We study a weighted empirical risk minimization (ERM) approach for integrating surrogate data into training. We analyze mathematically this method under several classical statistical models, and validate our findings empirically on datasets from different domains. Our main findings are: $(i)$ Integrating surrogate data can significantly reduce the test error on the original distribution. Surprisingly, this can happen even when the surrogate data is unrelated to the original ones. We trace back this behavior to the classical Stein's paradox. $(ii)$ In order to reap the benefit of surrogate data, it is crucial to use optimally weighted ERM. $(iii)$ The test error of models trained on mixtures of real and surrogate data is approximately described by a scaling law. This scaling law can be used to predict the optimal weighting scheme, and to choose the amount of surrogate data to add.
Neural Scaling Laws of Deep ReLU and Deep Operator Network: A Theoretical Study
Department of Mathematics, Hong Kong Baptist University（香港 Baptist 大学数学系）；Department of ACMS, University of Notre Dame（Notre Dame 大学ACMS系）；School of Mathematics, Georgia Institute of Technology（佐治亚理工学院数学系）；Department of Mathematics, UCLA（加州大学洛杉矶分校数学系）
Institut für Theoretische Physik, Universität Heidelberg（海德堡大学理论物理研究所）；Department of Physics, Harvard University（哈佛大学物理系）；LPNHE, Sorbonne Université, Université Paris Cité, CNRS/IN2P3（巴黎大学LPNHE）
Compute-optimal data scaling for neural surrogates via multi-fidelity training
Institute for Machine Learning, JKU Linz（林茨约翰内斯·开普勒大学机器学习研究所）；Mistral AI（Mistral AI公司）；Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
On the origin of neural scaling laws: from random graphs to natural language
Meta Superintelligence Labs, FAIR（Meta超智能实验室，FAIR）；Department of Physics, University of Maryland, College Park（大学物理系，马里兰大学College Park分校）；Joint Quantum Institute（联合量子研究所）
Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs
Meituan Inc.（美团公司）；The University of Hong Kong（香港大学）；Pennsylvania State University（宾夕法尼亚州立大学）；Peking University（北京大学）；The Hong Kong University of Science and Technology（香港科学与技术大学）
Reproducible scaling laws for contrastive language-image learning
LAION；UC Berkeley（加州大学伯克利分校）；HuggingFace；University of Washington（华盛顿大学）；Juelich Supercomputing Center (JSC)（于利希超级计算中心）；Research Center Juelich (FZJ)（于利希研究中心）
Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruning
The School of Computing and Information Systems, the University of Melbourne, Melbourne, Australia（墨尔本大学计算与信息系统学院）；Royal Melbourne Institute of Technology, Melbourne, Australia（皇家墨尔本理工学院）；Now at Google Australia（现就职于谷歌澳大利亚）
A Structure-Preserving Graph Neural Solver for Parametric Hyperbolic Conservation Laws
School of Artificial Intelligence；Suzhou Institute for Advanced Research, University of Science；Suzhou Big Data \& AI Research；Engineering Center, China；School of Mathematical Sciences, University of Science；School of Mathematical Sciences
