神经电路图:用于深度学习架构的沟通、实现和分析的稳健图表
Neural Circuit Diagrams: Robust Diagrams for the Communication, Implementation, and Analysis of Deep Learning Architectures
AI总结:
本文提出神经电路图,一种图形语言,用于精确沟通深度学习架构,解决现有方法无法同时表达轴细节与数据自由排列的问题,并展示其在多种架构及反向传播分析中的实用性。
摘要:
图表很重要。不幸的是,深度学习社区没有用于绘制架构图的标准方法。当前线性代数符号与临时图表的组合未能提供理解架构所有细节所需的精度。然而,这些细节对于忠实的实现、数学分析、进一步创新和伦理保证至关重要。我提出了神经电路图,这是一种专为沟通深度学习架构需求而设计的图形语言。神经电路图自然地跟踪数据排列的变化,精确展示操作如何在轴上广播,并显示线性操作的关键并行行为。现有图表方法的一个长期问题是无法同时表达轴的细节和数据的自由排列,而神经电路图解决了这一问题。它们的组合结构与代码类似,在图表和实现之间建立了紧密的对应关系。在这项工作中,我为机器学习研究人员介绍了神经电路图。在介绍神经电路图之后,我涵盖了一系列架构以展示其实用性并培养熟悉度。这包括Transformer架构、卷积(及其难以解释的扩展)、残差网络、U-Net和视觉Transformer。我附带了一个Jupyter笔记本,为图表与代码之间的紧密对应提供了证据。最后,我使用神经电路图检查了反向传播。我展示了它们在提供数学洞察和分析算法时间与空间复杂度方面的实用性。
英文摘要:
Diagrams matter. Unfortunately, the deep learning community has no standard method for diagramming architectures. The current combination of linear algebra notation and ad-hoc diagrams fails to offer the necessary precision to understand architectures in all their detail. However, this detail is critical for faithful implementation, mathematical analysis, further innovation, and ethical assurances. I present neural circuit diagrams, a graphical language tailored to the needs of communicating deep learning architectures. Neural circuit diagrams naturally keep track of the changing arrangement of data, precisely show how operations are broadcast over axes, and display the critical parallel behavior of linear operations. A lingering issue with existing diagramming methods is the inability to simultaneously express the detail of axes and the free arrangement of data, which neural circuit diagrams solve. Their compositional structure is analogous to code, creating a close correspondence between diagrams and implementation. In this work, I introduce neural circuit diagrams for an audience of machine learning researchers. After introducing neural circuit diagrams, I cover a host of architectures to show their utility and breed familiarity. This includes the transformer architecture, convolution (and its difficult-to-explain extensions), residual networks, the U-Net, and the vision transformer. I include a Jupyter notebook that provides evidence for the close correspondence between diagrams and code. Finally, I examine backpropagation using neural circuit diagrams. I show their utility in providing mathematical insight and analyzing algorithms' time and space complexities.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。