arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.01833cs.LG
收藏

理解正弦神经网络

Understanding Sinusoidal Neural Networks

IMPA(纯数学与应用数学国家研究所)

Tiago Novello

AI总结:

本文研究了正弦MLP的结构与表示能力,通过将其展开为谐波和揭示了其产生新频率的机制,并提供了振幅控制与截断方法,为网络的紧凑性提供了理论证明及训练和初始化的应用。

摘要:

在这项工作中,我们研究了正弦MLP(使用正弦作为激活函数的多层感知器网络)的结构和表示能力。这些神经网络(被称为神经场)在表示计算机图形学中的常见信号(如图像、符号距离函数和辐射场)方面已成为基础。这一成功主要归功于正弦MLP的两个关键特性:平滑性和紧凑性。这些函数是平滑的,因为它们是由仿射映射与正弦函数复合而成的。本工作提供了理论结果来证明正弦MLP的紧凑性属性,并提供了这些网络在定义和训练中的控制机制。\n我们提出通过将正弦MLP展开为谐波和来进行研究。首先,我们观察到它的第一层可以看作一个谐波字典,我们称之为输入正弦神经元。然后,隐藏层使用仿射映射组合该字典,并使用正弦调制输出,这产生了一个特殊的正弦神经元字典。我们证明,每个这样的正弦神经元都展开为一个谐波和,产生大量新频率,这些新频率表示为输入频率的整数线性组合。因此,每个隐藏神经元产生相同的频率,相应的振幅完全由隐藏的仿射映射决定。我们还提供了这些振幅的上界和一种排序方法,可以控制由此产生的近似,从而允许我们截断相应的级数。最后,我们展示了正弦MLP训练和初始化的应用。此外,我们表明如果输入神经元是周期性的,那么整个网络将是周期性的且具有相同的周期。我们将这些周期性网络与傅里叶级数表示联系起来。

英文摘要:

In this work, we investigate the structure and representation capacity of sinusoidal MLPs - multilayer perceptron networks that use sine as the activation function. These neural networks (known as neural fields) have become fundamental in representing common signals in computer graphics, such as images, signed distance functions, and radiance fields. This success can be primarily attributed to two key properties of sinusoidal MLPs: smoothness and compactness. These functions are smooth because they arise from the composition of affine maps with the sine function. This work provides theoretical results to justify the compactness property of sinusoidal MLPs and provides control mechanisms in the definition and training of these networks. We propose to study a sinusoidal MLP by expanding it as a harmonic sum. First, we observe that its first layer can be seen as a harmonic dictionary, which we call the input sinusoidal neurons. Then, a hidden layer combines this dictionary using an affine map and modulates the outputs using the sine, this results in a special dictionary of sinusoidal neurons. We prove that each of these sinusoidal neurons expands as a harmonic sum producing a large number of new frequencies expressed as integer linear combinations of the input frequencies. Thus, each hidden neuron produces the same frequencies, and the corresponding amplitudes are completely determined by the hidden affine map. We also provide an upper bound and a way of sorting these amplitudes that can control the resulting approximation, allowing us to truncate the corresponding series. Finally, we present applications for training and initialization of sinusoidal MLPs. Additionally, we show that if the input neurons are periodic, then the entire network will be periodic with the same period. We relate these periodic networks with the Fourier series representation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑