arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.05337cs.LGcs.CV
收藏

更大的图像分类器会记忆什么?

What do larger image classifiers memorise?

Google(谷歌)

Michal Lukasik, Vaishnavh Nagarajan, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar

AI总结:

该研究针对更大的图像分类模型是否记忆更多内容的问题开展实证分析,发现不同规模模型下训练样本记忆轨迹多样,知识蒸馏可抑制记忆并提升泛化,揭示了其提升泛化的机制。

摘要:

现代神经网络的成功推动了对记忆与泛化之间关联的研究:过参数化模型尽管能够完美拟合(记忆)完全随机的标签,却仍能实现良好泛化。为深入研究这一问题,Feldman提出了一种量化单个训练样本记忆程度的指标,并通过实验计算了ResNet在图像分类基准上的相应记忆分布。尽管这让我们初步窥见了真实世界模型的记忆内容,但也留下了一个根本问题:更大的神经模型会记忆更多内容吗?我们针对这一问题在图像分类基准上开展了全面的实证分析。研究发现,在不同模型规模下,训练样本呈现出出乎意料的多样化记忆轨迹:大多数样本在更大模型下的记忆程度会降低,其余样本则呈现帽形或上升的记忆趋势。我们证明,多种Feldman记忆分数的代理指标均无法捕捉这些根本趋势。最后,我们发现知识蒸馏(一种有效且流行的模型压缩技术)往往会抑制记忆,同时还能提升泛化能力。具体而言,记忆抑制主要作用于具有上升记忆轨迹的样本,这也揭示了蒸馏提升泛化能力的机制。

英文摘要:

The success of modern neural networks has prompted study of the connection between memorisation and generalisation: overparameterised models generalise well, despite being able to perfectly fit (memorise) completely random labels. To carefully study this issue, Feldman proposed a metric to quantify the degree of memorisation of individual training examples, and empirically computed the corresponding memorisation profile of a ResNet on image classification bench-marks. While an exciting first glimpse into what real-world models memorise, this leaves open a fundamental question: do larger neural models memorise more? We present a comprehensive empirical analysis of this question on image classification benchmarks. We find that training examples exhibit an unexpectedly diverse set of memorisation trajectories across model sizes: most samples experience decreased memorisation under larger models, while the rest exhibit cap-shaped or increasing memorisation. We show that various proxies for the Feldman memorization score fail to capture these fundamental trends. Lastly, we find that knowledge distillation, an effective and popular model compression technique, tends to inhibit memorisation, while also improving generalisation. Specifically, memorisation is mostly inhibited on examples with increasing memorisation trajectories, thus pointing at how distillation improves generalisation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑