Butterfly-Net:基于卷积神经网络的最优函数表示
Butterfly-Net: Optimal Function Representation Based on Convolutional Neural Networks
摘要:
深度网络,特别是卷积神经网络(CNN),已成功应用于机器学习的各个领域以及其他科学和工程领域的挑战性问题。本文介绍了Butterfly-Net,一种具有结构化和稀疏交叉通道连接的低复杂度CNN,以及用于网络族的Butterfly初始化策略。理论分析表明,蝶形网对输入数据傅立叶表示的逼近能力随深度的增加而呈指数衰减。将蝶形网与全连接神经网络相结合,证明了一大类问题可以很好地近似,网络复杂度取决于有效频率带宽而不是输入维数。在我们的分析中,常规的CNN被作为一个特殊的案例。数值实验验证了泊松方程的傅里叶核和能量泛函近似的解析结果。此外,所有实验都支持Butterfly初始化的训练优于随机初始化的训练。此外,添加剩余的跨通道连接,虽然显著增加了参数数量,但并没有太大提高训练后的准确性,并且对数据分布更敏感。
英文摘要:
Deep networks, especially convolutional neural networks (CNNs), have been successfully applied in various areas of machine learning as well as to challenging problems in other scientific and engineering fields. This paper introduces Butterfly-Net, a low-complexity CNN with structured and sparse cross-channel connections, together with a Butterfly initialization strategy for a family of networks. Theoretical analysis of the approximation power of Butterfly-Net to the Fourier representation of input data shows that the error decays exponentially as the depth increases. Combining Butterfly-Net with a fully connected neural network, a large class of problems are proved to be well approximated with network complexity depending on the effective frequency bandwidth instead of the input dimension. Regular CNN is covered as a special case in our analysis. Numerical experiments validate the analytical results on the approximation of Fourier kernels and energy functionals of Poisson's equations. Moreover, all experiments support that training from Butterfly initialization outperforms training from random initialization. Also, adding the remaining cross-channel connections, although significantly increase the parameter number, does not much improve the post-training accuracy and is more sensitive to data distribution.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。