arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.02592stat.MLcs.LGstat.ME
收藏

稀疏诱导分类先验改进信息瓶颈鲁棒性

Sparsity-Inducing Categorical Prior Improves Robustness of the Information Bottleneck

Michigan State University(密歇根州立大学);Argonne National Laboratory(阿贡国家实验室);Google Research(谷歌研究院)

Anirban Samaddar, Sandeep Madireddy, Prasanna Balaprakash, Tapabrata Maiti, Gustavo de los Campos, Ian Fischer

摘要:

信息瓶颈框架提供了一种系统的方法来学习表示,压缩输入中的滋扰信息,并提取有关预测的语义上有意义的信息。然而,选择固定所有数据的维度的先验分布会限制这种方法用于学习鲁棒表示的灵活性。我们提出了一种新的稀疏诱导穗板分类先验,使用稀疏性作为一种机制,以提供灵活性,允许每个数据点学习自己的维度分布。此外,它提供了一种机制,用于学习潜在变量和稀疏度的联合分布,因此可以考虑潜在空间中的完全不确定性。通过在MNIST、CIFAR-10和ImageNet数据上使用分布内和分布外学习场景进行的一系列实验,我们表明,与传统的固定维先验以及文献中提出的潜变量模型的其他稀疏诱导机制相比,所提出的方法提高了准确性和鲁棒性。

英文摘要:

The information bottleneck framework provides a systematic approach to learning representations that compress nuisance information in the input and extract semantically meaningful information about predictions. However, the choice of a prior distribution that fixes the dimensionality across all the data can restrict the flexibility of this approach for learning robust representations. We present a novel sparsity-inducing spike-slab categorical prior that uses sparsity as a mechanism to provide the flexibility that allows each data point to learn its own dimension distribution. In addition, it provides a mechanism for learning a joint distribution of the latent variable and the sparsity and hence can account for the complete uncertainty in the latent space. Through a series of experiments using in-distribution and out-of-distribution learning scenarios on the MNIST, CIFAR-10, and ImageNet data, we show that the proposed approach improves accuracy and robustness compared to traditional fixed-dimensional priors, as well as other sparsity induction mechanisms for latent variable models proposed in the literature.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑