arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.01618cs.LGcs.NAmath.NA
收藏

算子学习与数值分析的结合:通过迭代方法改进神经网络

Operator Learning Meets Numerical Analysis: Improving Neural Networks through Iterative Methods

Idaho State University(爱达荷州立大学);Yale University(耶鲁大学);University of Michigan, Ann Arbor(密歇根大学安娜堡分校)

Emanuele Zappala, Daniel Levine, Sizhuang He, Syed Rizvi, Sacha Levy, David van Dijk

AI总结:

本研究将深度学习与经典数值分析类比,构建基于算子方程迭代方法的理论框架并给出收敛证明,验证迭代可提升网络性能,还提出迭代图神经网络PIGN,为深度学习提供理论支撑与设计指导。

摘要:

深度神经网络尽管在众多应用中取得成功,但往往缺乏成熟的理论基础。本文通过类比深度学习与经典数值分析,填补了这一空白。我们将神经网络构建为以不动点代表期望解的算子,由此建立了一个基于算子方程迭代方法的理论框架。在给定条件下,我们基于不动点理论给出了收敛性证明。我们证明,扩散模型、AlphaFold等主流架构本质上都采用了迭代算子学习。实证评估显示,通过网络算子执行迭代可提升性能。我们还提出了一种迭代图神经网络PIGN,进一步验证了迭代的优势。本研究旨在融合数值分析的见解,深化对深度学习的理解,有望为未来网络设计提供更清晰的理论支撑与更优的性能表现。

英文摘要:

Deep neural networks, despite their success in numerous applications, often function without established theoretical foundations. In this paper, we bridge this gap by drawing parallels between deep learning and classical numerical analysis. By framing neural networks as operators with fixed points representing desired solutions, we develop a theoretical framework grounded in iterative methods for operator equations. Under defined conditions, we present convergence proofs based on fixed point theory. We demonstrate that popular architectures, such as diffusion models and AlphaFold, inherently employ iterative operator learning. Empirical assessments highlight that performing iterations through network operators improves performance. We also introduce an iterative graph neural network, PIGN, that further demonstrates benefits of iterations. Our work aims to enhance the understanding of deep learning by merging insights from numerical analysis, potentially guiding the design of future networks with clearer theoretical underpinnings and improved performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑