arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.06756cs.LGcs.AI
收藏

超越神经网络特征相似性:网络特征复杂度及其基于范畴论的诠释

Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory

Shanghai Jiao Tong University(上海交通大学)

Yiting Chen, Zhanpeng Zhou, Junchi Yan

AI总结:

本文针对神经网络特征相似性描述模糊的问题,提出功能等价特征概念,定义特征复杂度,并基于范畴论进行形式化解释,同时设计 Iterative Feature Merging 算法量化特征复杂度,证明其可作为有效的数据无关模型剪枝方法。

摘要:

神经网络的行为仍然不透明,一个近期被广泛注意到的现象是,使用不同随机参数初始化的网络往往能达到相似的性能。这一现象在测量不同网络学习到的特征之间的相似性方面引起了显著关注。然而,由于几乎不存在等价特征,特征相似性在描述同一特征时可能显得模糊。在本文中,我们扩展了等价特征的概念,并给出了所谓功能等价特征的定义。这些特征在特定变换下产生等价的输出。利用这一定义,我们旨在针对神经网络每一层学习到的特征冗余性,推导出一个更本质的所谓特征复杂度度量。我们通过范畴论这一发展成熟的数学领域的视角,为我们的方法提供了形式化的解释。为了量化特征复杂度,我们进一步提出了一种名为 Iterative Feature Merging 的高效算法。我们的实验结果从多个视角验证了我们的想法和理论。我们经验性地证明了,功能等价性广泛存在于同一神经网络学习到的不同特征中,并且我们能够在不影响性能的情况下减少网络的参数量。IFM 作为一种与数据无关的模型剪枝方法展现出了巨大潜力。我们还得出了一些关于所定义特征复杂度的有趣经验发现。

英文摘要:

The behavior of neural networks still remains opaque, and a recently widely noted phenomenon is that networks often achieve similar performance when initialized with different random parameters. This phenomenon has attracted significant attention in measuring the similarity between features learned by distinct networks. However, feature similarity could be vague in describing the same feature since equivalent features hardly exist. In this paper, we expand the concept of equivalent feature and provide the definition of what we call functionally equivalent features. These features produce equivalent output under certain transformations. Using this definition, we aim to derive a more intrinsic metric for the so-called feature complexity regarding the redundancy of features learned by a neural network at each layer. We offer a formal interpretation of our approach through the lens of category theory, a well-developed area in mathematics. To quantify the feature complexity, we further propose an efficient algorithm named Iterative Feature Merging. Our experimental results validate our ideas and theories from various perspectives. We empirically demonstrate that the functionally equivalence widely exists among different features learned by the same neural network and we could reduce the number of parameters of the network without affecting the performance.The IFM shows great potential as a data-agnostic model prune method. We have also drawn several interesting empirical findings regarding the defined feature complexity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑