发表机构
Amadeus（阿马德乌斯）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究随机优化中样本级危害问题，提出将参数更新形式化为优化问题，引入高效代理，通过降维、限制优化层等方法减少内存和速度瓶颈，集成到标准优化器，实验证明能减少样本干扰、提升泛化能力。
AI 中文摘要
现代优化器将当前小批次的梯度与历史优化状态（如动量或自适应矩）相结合。虽然非常有效，但跨批次聚合并纳入此历史记录可能会产生增加单个样本损失的参数更新。我们将这种效应称为危害，并将参数更新形式化为一个优化问题，明确最小化批次平均和过去优化状态对当前数据的冲突影响。由于精确公式难以处理，我们引入了一个高效的代理。我们首先将问题的维度降低到批次大小，然后通过成功将优化限制在最后一个线性层来大幅减少内存和速度瓶颈。这取决于一个意外发现，即仅该层就能可靠地捕获样本级梯度的二阶统计量。由此产生的替代问题很容易集成到标准优化器（如SGD和AdamW）中，并且可以使用少量适合GPU的迭代来解决。至关重要的是，该方法具有良好的缩放特性，随着模型大小或输入的增加，相对计算开销会缩小。在图像分类基准上的实验证实了样本级干扰的减少和泛化能力的提高。
英文摘要
Modern optimizers combine gradients from the current mini-batch with historical optimization state, such as momentum or adaptive moments. While effective, this standard practice can produce parameter updates that actively increase the loss of individual samples. We term this phenomenon per-sample interference and propose redefining the parameter update as an optimization problem that explicitly minimizes it. Because the exact formulation of the problem is computationally prohibitive, we introduce a highly efficient surrogate. By reducing the problem's dimensionality to the batch size and restricting the optimization to the last linear layer, we overcome memory and speed bottlenecks. This strategy hinges on our unexpected finding that this layer alone can reliably capture core second-order statistics of the full network. The resulting surrogate problem integrates readily into standard optimizers like SGD and AdamW, and can be solved using a small number of GPU-friendly iterations. Crucially, the method exhibits favorable scaling properties, as the relative computational overhead shrinks as the model size or input grows. Experiments on image classification benchmarks confirm reduced per-sample interference and improved generalization.
Journal refProceedings of the 43rd International Conference on Machine Learning (ICML 2026)
