arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.01236cs.LGcs.CRq-bio.NC
收藏

神经网络中的等距表示提高鲁棒性

Isometric Representations in Neural Networks Improve Robustness

University of Oslo(奥斯陆大学)

Kosio Beshkov, Jonas Verhellen, Mikkel Elle Lepperød

摘要:

人工和生物代理可以在完全随机和非结构化的数据中学习。数据的结构被编码在数据点之间的度量关系中。在神经网络的上下文中,层内的神经元活动形成反映该层对其输入实现的变换的表示。为了以真实的方式利用数据中的结构,这种表示应该反映输入距离,因此是连续的和等距的。支持这一说法的是,神经科学的最新发现提出,泛化和鲁棒性与神经表征的连续可微性有关。在机器学习中,大多数算法缺乏鲁棒性,并且通常被认为依赖于与人类使用的数据不同的数据方面,这在对抗攻击中很常见。在交叉熵分类过程中,网络表示的度量和结构属性通常在类之间和类内都被破坏。训练的这种副作用可能会导致在未保留此类结构的位置附近的扰动下出现不稳定。获得鲁棒性的标准解决方案之一是添加特设正则化项,但据我们所知,强制表示保留输入数据的度量结构作为稳定机制尚未被研究。在这项工作中,我们训练神经网络进行分类,同时保持类内度量结构,导致等距类内表示。这样的网络表示有利于准确和鲁棒的推理。通过堆叠具有此属性的层,我们创建了一个网络架构,该架构有助于内部神经表征的分层操作。最后,我们验证了等距正则化提高了MNIST对对抗攻击的鲁棒性。

英文摘要:

Artificial and biological agents cannon learn given completely random and unstructured data. The structure of data is encoded in the metric relationships between data points. In the context of neural networks, neuronal activity within a layer forms a representation reflecting the transformation that the layer implements on its inputs. In order to utilize the structure in the data in a truthful manner, such representations should reflect the input distances and thus be continuous and isometric. Supporting this statement, recent findings in neuroscience propose that generalization and robustness are tied to neural representations being continuously differentiable. In machine learning, most algorithms lack robustness and are generally thought to rely on aspects of the data that differ from those that humans use, as is commonly seen in adversarial attacks. During cross-entropy classification, the metric and structural properties of network representations are usually broken both between and within classes. This side effect from training can lead to instabilities under perturbations near locations where such structure is not preserved. One of the standard solutions to obtain robustness is to add ad hoc regularization terms, but to our knowledge, forcing representations to preserve the metric structure of the input data as a stabilising mechanism has not yet been studied. In this work, we train neural networks to perform classification while simultaneously maintaining within-class metric structure, leading to isometric within-class representations. Such network representations turn out to be beneficial for accurate and robust inference. By stacking layers with this property we create a network architecture that facilitates hierarchical manipulation of internal neural representations. Finally, we verify that isometric regularization improves the robustness to adversarial attacks on MNIST.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑