arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.16845stat.MLcs.LGmath.OC
收藏

有限训练预算下的特征归一化

Feature Normalization under Finite Training Budgets

University of Copenhagen(哥本哈根大学)

Johan Larsson

AI总结:

该研究针对有限训练预算,对比标准化与未缩放训练的稀有子群风险,推导相关公式,明确不同信号族下的最优方法,还分析了观测复用等场景下的排序变化。

摘要:

标准化稀有特征可帮助随机梯度下降在有限预算内学习其效应,但也会改变学习的噪声成本。我们在相同观测上限下,当每种方法选择非负常数学习率和确定性停止时间时,刻画了标准化训练与未缩放训练可达到的最低稀有子群风险。对于两个独立的二元特征,当稀有特征的流行度消失但稀有观测的期望数量保持有限时,我们推导了两种方法的优化风险。对学习率和停止时间的统一控制将每个极限神谕简化为标量最小化。在存在正噪声的情况下,当将信号转移到稀有特征时,首选的缩放方式恰好改变一次,而初始化时的极限风险保持固定为正。这些公式还解释了噪声和预算如何影响比较。单独的有限流行度边界表明,在稀有主导信号族上标准化更优,在常见主导信号族上未缩放训练更优。结果涉及具有总体预处理和新鲜观测的最终迭代。精确矩搜索评估了有限流行度下的近似情况。在一项受控研究中,当评估前调整选择固定时,两种排序在观测复用、估计预处理和截距拟合下均保持。在观测训练后选择预测器可反转排序,即使选择使用精确总体风险。

英文摘要:

Standardizing a rare feature can help stochastic gradient descent learn its effect within a finite budget, but it also changes the noise cost of learning. We characterize the lowest rare-subgroup risks attainable by standardized and unscaled training when each chooses a nonnegative constant learning rate and a deterministic stopping time under the same observation cap. For two independent binary features, we derive both optimized risks as prevalence vanishes while the expected number of rare observations remains finite. Uniform control over rates and stopping times reduces each limiting oracle to a scalar minimization. With positive noise, the preferred scaling changes exactly once as we move signal to the rare feature while the limiting risk at initialization stays fixed and positive. The formulas also explain how noise and budget affect the comparison. Separate finite-prevalence bounds favor standardization on a rare-dominant signal family and unscaled training on a common-dominant family. The results concern final iterates with population preprocessing and fresh observations. Exact-moment searches assess the approximation at finite prevalence. In a controlled study, both orderings persist under observation reuse, estimated preprocessing, and intercept fitting when tuning choices are fixed before evaluation. Selecting a predictor after observing training can reverse an ordering, even when selection uses exact population risk.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑