Deep neural networks, despite their success in numerous applications, often function without established theoretical foundations. In this paper, we bridge this gap by drawing parallels between deep learning and classical numerical analysis. By framing neural networks as operators with fixed points representing desired solutions, we develop a theoretical framework grounded in iterative methods for operator equations. Under defined conditions, we present convergence proofs based on fixed point theory. We demonstrate that popular architectures, such as diffusion models and AlphaFold, inherently employ iterative operator learning. Empirical assessments highlight that performing iterations through network operators improves performance. We also introduce an iterative graph neural network, PIGN, that further demonstrates benefits of iterations. Our work aims to enhance the understanding of deep learning by merging insights from numerical analysis, potentially guiding the design of future networks with clearer theoretical underpinnings and improved performance.
Efficiently improving key weather variables forecasting by performing the guided iterative prediction in latent space
Wuhan University(武汉大学);Hubei Key Laboratory of Quantitative Remote Sensing of Land and Atmosphere(湖北省陆气定量遥感重点实验室);School of Remote Sensing and Information Engineering(遥感信息工程学院);Hubei Luojia Laboratory(湖北珞珈实验室)
Deep Learning Meets Teleconnections: Improving S2S Predictions for European Winter Weather
TU Berlin(柏林工业大学);ATB(莱布尼茨农业工程与生物经济研究所);Leipzig University(莱比锡大学);University of Reading(雷丁大学);The Alan Turing Institute(阿兰·图灵研究所);University of Potsdam(波茨坦大学);BIFOLD – Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究所)
Self-Improving Tabular Language Models via Iterative Reward-Guided Post-Training
Department of Engineering, University of Cambridge(剑桥大学工程系);CISPA Helmholtz Center for Information Security, Saarbrücken, Germany(德国萨尔布吕肯信息安全中心);The Alan Turing Institute, London(伦敦阿兰·图灵研究所)
School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院);Faculty of Information Technology, Macau University of Science and Technology(澳门科技大学信息技术学院);State key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications(北京邮电大学网络与交换技术国家重点实验室)