烧蚀数据增强带来的损失
The Penalty Imposed by Ablated Data Augmentation
摘要：
有一组数据增强技术可以随机消融部分输入。这些包括输入丢失、切断和随机擦除。我们称这些技术为消融数据增强。虽然这些技术在精神上看起来相似，并且在各种领域都成功地提高了模型性能，但我们还没有像对L1或L2等其他正则化技术那样对这些技术之间的差异有数学上的理解。首先，我们研究了线性回归的平均消融数据增强和倒辍学的形式化模型。我们证明，消融数据增强相当于优化普通最小二乘目标，以及一个惩罚，我们称之为贡献协方差惩罚和倒dropout，一个更常见的实现比流行的框架中的dropout，相当于优化普通最小二乘目标以及修改L2。对于深度网络，如果我们将贡献替换为属性，将系数替换为平均梯度，我们将证明结果的经验版本，即，贡献协方差惩罚和修正的L2惩罚随着各种网络上相应的消融数据增强的增加而下降。
英文摘要：
There is a set of data augmentation techniques that ablate parts of the input at random. These include input dropout, cutout, and random erasing. We term these techniques ablated data augmentation. Though these techniques seems similar in spirit and have shown success in improving model performance in a variety of domains, we do not yet have a mathematical understanding of the differences between these techniques like we do for other regularization techniques like L1 or L2. First, we study a formal model of mean ablated data augmentation and inverted dropout for linear regression. We prove that ablated data augmentation is equivalent to optimizing the ordinary least squares objective along with a penalty that we call the Contribution Covariance Penalty and inverted dropout, a more common implementation than dropout in popular frameworks, is equivalent to optimizing the ordinary least squares objective along with Modified L2. For deep networks, we demonstrate an empirical version of the result if we replace contributions with attributions and coefficients with average gradients, i.e., the Contribution Covariance Penalty and Modified L2 Penalty drop with the increase of the corresponding ablated data augmentation across a variety of networks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
