通过倾斜指数层提升鲁棒性:通信理论视角
Improving Robustness via Tilted Exponential Layer: A Communication-Theoretic Perspective
AI总结:
本文受通信理论启发提出TEXP层,通过神经元竞争、倾斜指数目标和倾斜softmax推理提升网络对噪声与常见损坏的鲁棒性,并可与数据增强结合获得累积增益。
摘要:
用于增强深度网络鲁棒性的最先进技术大多依赖于经验风险最小化,并配合适当的数据增强。本文提出一种受通信理论启发的互补方法,旨在通过学习和推理期间的神经竞争来提高神经网络层输出端的信噪比。除标准经验风险最小化外,神经元还通过最大化该层的倾斜指数(tilted exponential,TEXP)目标函数,竞争以稀疏表示层输入。TEXP学习可解释为在数据噪声服从高斯模型的情况下对匹配滤波器进行最大似然估计。TEXP层中的推理通过用倾斜softmax替代批归一化来完成,这可解释为计算每个神经元所表示的相互竞争的信号假设的后验概率。在通过简化模型给出洞见之后,我们在标准图像数据集上的实验表明,TEXP学习和推理可增强对噪声及其他常见损坏的鲁棒性,而无需数据增强。通过将TEXP与数据增强技术适当结合,还可在应对这一系列失真时获得进一步的累积鲁棒性增益。我们所有实验的代码可在https://github.com/bhagyapuranik/texp_for_robustness获取。
英文摘要:
State-of-the-art techniques for enhancing robustness of deep networks mostly rely on empirical risk minimization with suitable data augmentation. In this paper, we propose a complementary approach motivated by communication theory, aimed at enhancing the signal-to-noise ratio at the output of a neural network layer via neural competition during learning and inference. In addition to standard empirical risk minimization, neurons compete to sparsely represent layer inputs by maximization of a tilted exponential (TEXP) objective function for the layer. TEXP learning can be interpreted as maximum likelihood estimation of matched filters under a Gaussian model for data noise. Inference in a TEXP layer is accomplished by replacing batch norm by a tilted softmax, which can be interpreted as computation of posterior probabilities for the competing signaling hypotheses represented by each neuron. After providing insights via simplified models, we show, by experimentation on standard image datasets, that TEXP learning and inference enhances robustness against noise and other common corruptions, without requiring data augmentation. Further cumulative gains in robustness against this array of distortions can be obtained by appropriately combining TEXP with data augmentation techniques. The code for all our experiments is available at https://github.com/bhagyapuranik/texp_for_robustness.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。