arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.05641cs.LG
收藏

基于蒸馏的广义界

Generalization bounds via distillation

Columbia University(哥伦比亚大学);University of Illinois(伊利诺伊大学)

Daniel Hsu, Ziwei Ji, Matus Telgarsky, Lan Wang

摘要:

本文从理论上研究了以下经验现象:给定一个具有较差泛化边界的高复杂度网络,可以将其提取为具有几乎相同预测但复杂度较低且泛化边界小得多的网络。主要贡献是分析表明,原始网络继承了这种良好的泛化范围,从它的蒸馏,假设使用良好的数据增强。这个界限以抽象和具体的形式呈现,后者由简化技术补充,以处理具有卷积层,全连接层和跳过连接的现代计算图,仅举几例。为了完成这个故事,一个(松散的)经典的一致收敛分析压缩,以及各种实验cifar和mnist证明类似的泛化性能之间的原始网络和它的蒸馏。

英文摘要:

This paper theoretically investigates the following empirical phenomenon: given a high-complexity network with poor generalization bounds, one can distill it into a network with nearly identical predictions but low complexity and vastly smaller generalization bounds. The main contribution is an analysis showing that the original network inherits this good generalization bound from its distillation, assuming the use of well-behaved data augmentation. This bound is presented both in an abstract and in a concrete form, the latter complemented by a reduction technique to handle modern computation graphs featuring convolutional layers, fully-connected layers, and skip connections, to name a few. To round out the story, a (looser) classical uniform convergence analysis of compression is also presented, as well as a variety of experiments on cifar and mnist demonstrating similar generalization performance between the original network and its distillation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑