arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.19013cs.LG
收藏

图像自蒸馏的数据密度缩放定律

Data Density Scaling Laws for Image Self-Distillation

YerevaNN(耶列瓦恩);Yerevan State University(埃里温国立大学)

Alvard Barseghyan, Ani Vanyan, Hakob Tamazyan, Hrant Khachatrian

AI总结:

本研究针对图像自蒸馏模型检验现有缩放定律假设,发现样本重复影响具数据集依赖性,提出改进的Chinchilla式缩放定律,还探究了子集选择与增强对性能的影响。

摘要:

当前的缩放定律公式表明,增加数据集中的唯一样本数量可提升模型性能,而重复接触相同样本的价值会逐渐降低。本研究针对图像自蒸馏模型检验该假设,通过在Maxar、SATLAS、Waymo、nuPlan、Ego4D、ImageNet、YFCC15M等不同数据集上开展系统实验,证实样本重复对下游性能的影响并非微不足道,且具有数据集依赖性。我们发现,在某些场景下,对较小的随机子集重复训练的性能可优于对更大的唯一样本集训练的性能,这一发现凸显了现有缩放定律的局限性。为填补该缺口,我们提出了改进的Chinchilla式缩放定律,该公式明确考虑两个因素:唯一样本数量及每个样本的重复次数。此公式为理解数据-计算权衡提供了框架,并在重复起核心作用的部分场景中提升了预测效果,不过其益处并非在所有数据集上一致。最后,我们探究了子集选择与数据增强对下游性能的影响。

英文摘要:

Current scaling law formulations suggest that increasing the number of unique samples in a dataset consistently improves model performance, while repeated exposure to the same samples provides diminishing value. In this work, we challenge this assumption for image self-distillation models. We train iBOT models on random subsets of seven datasets spanning satellite imagery, driving scenes, egocentric video, and natural images, and evaluate their frozen representations on downstream tasks. We propose a density-interaction scaling law that combines unique-image count $U$ and repetition $R$, fitting it on runs with at most 50 million image presentations and testing it at larger budgets $C$. On Maxar dataset, a random subset containing approximately 1.5% of the training pool improves average nearest-neighbor classification accuracy by 3.11 percentage points over training on the full pool at the same budget of 200 million image presentations. Smaller subsets also outperform the full pool on SATLAS and Ego4D, whereas ImageNet favors larger pools at higher budgets. At a fixed subset size, further training can improve performance, lead to a plateau, or reduce performance. Our law captures smaller-subset advantages that the compared baselines cannot represent and improves prediction at higher budgets on Maxar, SATLAS, and Ego4D under nearest-neighbor evaluation. These findings show that both the number of unique images and how often they are repeated matter when allocating a training budget.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑