上采样还是上加权？在严重不平衡数据集上的平衡训练
Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets
AI总结：
本文从理论和实证上分析了多语言不平衡训练中上采样与上加权在随机梯度下降下的差异，并提出 Cooldown 策略，通过逐步降低上采样程度兼顾收敛速度与防过拟合。
摘要：
不同领域的数据丰度呈现长尾分布：少数领域拥有丰富数据，而大多数领域面临数据稀缺。我们的工作聚焦于多语言场景，其中可用数据严重偏向高资源语言。解决这一差距的两种常见策略是对低资源数据进行上采样（温度采样）和对低资源损失进行上加权（标量化）。这些方法通常被认为等价，但这种等价性尚未得到严格证明，这促使我们开展研究。通过理论和实证分析，我们确定了这两种方法何时等价以及何时出现分歧。我们证明，它们在完全梯度下降下是等价的，但在随机梯度下降下由于梯度方差的差异而不同。具体而言，与标量化相比，温度采样在梯度估计中表现出更低的方差，从而收敛更快，但过拟合风险更高。基于这些见解，我们提出了 Cooldown，该策略从对低资源语言进行大幅上采样开始以加速收敛，并逐渐降低上采样程度以防止过拟合——实现两全其美。我们的方法与现有的数据重加权技术相比具有竞争力，同时提供计算效率。
英文摘要：
Data abundance across different domains exhibits a long-tailed distribution: few domains have abundant data, while most face data scarcity. Our work focuses on a multilingual setting, where available data is heavily skewed towards high-resource languages. Two common strategies to address this disparity are upsampling low-resource data (Temperature Sampling) and upweighting low-resource loss (Scalarization). These methods are often assumed to be equivalent, but this equivalence has not been rigorously established, prompting our investigation. Through theoretical and empirical analysis, we identify when these two methods are equivalent and when they diverge. We prove that they are equivalent under full gradient descent but differ under stochastic gradient descent due to differences in gradient variance. Specifically, Temperature Sampling exhibits lower variance in gradient estimation compared to Scalarization, leading to faster convergence but a higher risk of overfitting. Based on these insights, we propose Cooldown, a strategy that starts by heavily upsampling low-resource languages to accelerate convergence and gradually reduces the upsampling to prevent overfitting -- achieving the best of both worlds. Our method competes effectively with existing data re-weighting techniques while offering computational efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
