arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.12780cs.LG
收藏

用于深度权重空间学习的等变架构

Equivariant Architectures for Learning in Deep Weight Spaces

Bar-Ilan University(巴伊兰大学);Nvidia(英伟达)

Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, Haggai Maron

AI总结:

本文提出一种对 MLP 权重的自然神经元置换对称等变的深度权重空间学习架构,完整刻画仿射等变与不变层,并验证其优于基线。

摘要:

设计用于处理原始权重矩阵形式神经网络的机器学习架构,是一个新近提出的研究方向。遗憾的是,深度权重空间独特的对称结构使这一设计极具挑战性。若能成功,此类架构将能够执行一系列引人关注的任务,从使预训练网络适应新领域,到编辑以函数形式表示的对象(INR 或 NeRF)。作为迈向这一目标的第一步,我们在此提出一种用于在深度权重空间中学习的新型网络架构。它将预训练 MLP 的权重和偏置的拼接作为输入,并通过一组对 MLP 权重的自然置换对称具有等变性的层的组合来处理该输入:改变 MLP 中间层神经元的顺序不会影响其所表示的函数。我们完整刻画了针对这些对称的所有仿射等变层和不变层,并展示了如何使用三种基本操作来实现这些层:池化、广播,以及以适当方式作用于输入的全连接层。我们在多种学习任务中证明了所提架构的有效性及其相对于自然基线的优势。

英文摘要:

Designing machine learning architectures for processing neural networks in their raw weight matrix form is a newly introduced research direction. Unfortunately, the unique symmetry structure of deep weight spaces makes this design very challenging. If successful, such architectures would be capable of performing a wide range of intriguing tasks, from adapting a pre-trained network to a new domain to editing objects represented as functions (INRs or NeRFs). As a first step towards this goal, we present here a novel network architecture for learning in deep weight spaces. It takes as input a concatenation of weights and biases of a pre-trained MLP and processes it using a composition of layers that are equivariant to the natural permutation symmetry of the MLP's weights: Changing the order of neurons in intermediate layers of the MLP does not affect the function it represents. We provide a full characterization of all affine equivariant and invariant layers for these symmetries and show how these layers can be implemented using three basic operations: pooling, broadcasting, and fully connected layers applied to the input in an appropriate manner. We demonstrate the effectiveness of our architecture and its advantages over natural baselines in a variety of learning tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑