arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.14290cs.LG
收藏

深度神经网络中的逐层输入可恢复性

Layer-wise Input Recoverability in Deep Neural Networks

University of California, Irvine(加州大学欧文分校)

Maxwell Spiridonov, Alireza Rahmansetayesh, Pierre Baldi

AI总结:

该研究针对CNN提出基于输入块的可恢复性评估方法,通过单属性随机控制探究预训练属性对可恢复性的影响,发现复制预训练奇异向量可提升可恢复性,复制奇异值谱或激活稀疏性则降低可恢复性。

摘要:

深度神经网络的内部工作机制仍未被充分理解。信息瓶颈框架通过隐藏表示与网络输入之间的互信息来表征深度网络,已被用于识别输入图像相对于输出标签的输入信息压缩情况。我们转而研究卷积神经网络(CNN)中,从某层的输出可恢复该层自身输入的程度。卷积层每次将其滤波器应用于一个输入块,因此我们基于采样的输入块,使用基于重构的探针评估可恢复性,而非针对整个特征图。探针的大小因此取决于通道数和核大小,而非特征图大小,这使得我们的方法可扩展到具有宽特征图的深度CNN。对于每一层,我们使用线性解码器和线性加残差多层感知机(linear-plus-residual-MLP)解码器,比较预训练、随机和重构优化的编码器。我们将可恢复性度量为输入块与它们在保留集上的重构之间的R²。在9个ImageNet CNN上,可恢复性通常随深度增加而降低,但更一致地由该层的压缩性预测。学习属性的可恢复性(预训练编码器与随机编码器之间的差异)在不同架构间并不均匀。我们引入单属性随机控制,将一个预训练属性迁移到随机编码器上,发现每个控制都会以跨模型一致的符号改变可恢复性:复制预训练奇异向量往往会提高可恢复性,而复制预训练奇异值谱或激活稀疏性则会降低可恢复性。

英文摘要:

The inner workings of deep neural networks are still not well understood. The information bottleneck framework, which characterizes deep networks by the mutual information between their hidden representations and the network's input, has been used to identify compression of input information with respect to an input image and output labels. We instead study how much of a layer's own input is recoverable from the layer's output in convolutional neural networks (CNNs). A convolutional layer applies its filters to one input patch at a time, so we evaluate recoverability on sampled input patches with a reconstruction-based probe and not on the entire feature map. The probe's size thus depends on channels and kernel size and not on feature map size, which lets our method scale to deep CNNs with wide feature maps. For each layer, we compare pretrained, random, and reconstruction-optimized encoders, using linear and linear-plus-residual-MLP decoders. We measure recoverability as $R^2$ between input patches and their reconstructions on a held-out set. Across nine ImageNet CNNs, recoverability generally decreases with depth but is more consistently predicted by the compression of the layer. Learning-attributed recoverability, the difference between pretrained and random encoders, is not uniform across architectures. We introduce single-property random controls, which transfer one pretrained property onto a random encoder, and find that each control changes recoverability with a consistent sign across models: copying pretrained singular vectors tends to raise recoverability, while copying the pretrained singular-value spectrum or activation sparsity lowers it.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑