arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2002.10006cs.LGstat.ML
收藏

关于超网络的模块性

On the Modularity of Hypernetworks

Facebook AI Research (FAIR)(脸书人工智能研究院(FAIR));Tel Aviv University(特拉维夫大学)

Tomer Galanti, Lior Wolf

摘要:

在学习将输入$I$映射到函数$h_I:\mathcal{X}\to \mathbb{R}$的上下文中,比较两种替代方法:(i)基于嵌入的方法,其学习固定函数,其中$I$被编码为调节信号$e(I)$,并且所学习的函数采取$h_I(x) = q(x,e(I))$的形式,以及(ii)超网络,其中函数$h_I(x) = g(x;θ_I)$的权重$θ_I$由超网络$f$给出为$θ_I=f(I)$。在本文中,我们将模块性的属性定义为针对每个输入实例$I$有效地学习不同函数的能力。为了这个目的,我们采用了一个表达的角度来看,这个属性和扩展的理论Devore等。1996年,并提供了一个下界的复杂性(可训练参数的数量)的神经网络函数逼近器,通过消除近似方法的要求是强大的。然后,我们的结果被用来比较$q$和$g$的复杂性,表明在一定的条件下,当让功能$e$和$f$是大,因为我们希望,$g$可以小于$q$的数量级。与基于嵌入的方法相比,这揭示了超网络的模块化。此外,我们表明,对于一个结构化的目标函数,在一个超网络的可训练参数的总数是小于一个标准的神经网络和嵌入方法的可训练参数的数量级。

英文摘要:

In the context of learning to map an input $I$ to a function $h_I:\mathcal{X}\to \mathbb{R}$, two alternative methods are compared: (i) an embedding-based method, which learns a fixed function in which $I$ is encoded as a conditioning signal $e(I)$ and the learned function takes the form $h_I(x) = q(x,e(I))$, and (ii) hypernetworks, in which the weights $θ_I$ of the function $h_I(x) = g(x;θ_I)$ are given by a hypernetwork $f$ as $θ_I=f(I)$. In this paper, we define the property of modularity as the ability to effectively learn a different function for each input instance $I$. For this purpose, we adopt an expressivity perspective of this property and extend the theory of Devore et al. 1996 and provide a lower bound on the complexity (number of trainable parameters) of neural networks as function approximators, by eliminating the requirements for the approximation method to be robust. Our results are then used to compare the complexities of $q$ and $g$, showing that under certain conditions and when letting the functions $e$ and $f$ be as large as we wish, $g$ can be smaller than $q$ by orders of magnitude. This sheds light on the modularity of hypernetworks in comparison with the embedding-based method. Besides, we show that for a structured target function, the overall number of trainable parameters in a hypernetwork is smaller by orders of magnitude than the number of trainable parameters of a standard neural network and an embedding method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑