采用直接反馈对齐的大规模现代深度学习架构的简化光学训练
Streamlined optical training of large-scale modern deep learning architectures with direct feedback alignment
AI总结：
本研究在混合电光平台上实验实现直接反馈对齐训练算法，以低功耗高速完成大规模随机矩阵乘法，实现超10亿参数现代架构的光学训练，为AI增长提供新路径。
摘要：
现代深度学习几乎完全依赖专用的电子硬件加速器。具有低功耗、高运算速度的光子学方法正越来越多地被用于推理，但截至目前，其应用大多仍局限于相对基础的任务。与此同时，深度复杂神经网络的训练问题——目前绝大多数通过反向传播完成——仍是当前架构规模及相应性能的重要限制因素，也是主要的计算与能耗瓶颈。在本研究中，我们在一个混合电光平台上实验实现了一种通用且可扩展的训练算法，名为direct feedback alignment（直接反馈对齐）。一个光学处理单元执行该算法的核心运算——大规模随机矩阵乘法，在30瓦以下的功耗下运算速度可达1500 TeraOPS。我们对包含Transformers在内、参数量超过10亿的现代深度学习架构进行光学训练，在语言、视觉以及基于扩散的生成任务上均取得了良好性能。我们研究了训练时间的缩放规律，并证明了我们的混合电光方法在超深、超宽神经网络方面的潜在优势，从而为在传统冯·诺依曼方法之外维持现代人工智能的指数级增长开辟了一条颇具前景的路径。
英文摘要：
Modern deep learning relies nearly exclusively on dedicated electronic hardware accelerators. Photonic approaches, with low consumption and high operation speed, are increasingly considered for inference but, to date, remain mostly limited to relatively basic tasks. Simultaneously, the problem of training deep and complex neural networks, overwhelmingly performed through backpropagation, remains a significant limitation to the size and, consequently, the performance of current architectures and a major compute and energy bottleneck. Here, we experimentally implement a versatile and scalable training algorithm, called direct feedback alignment, on a hybrid electronic-photonic platform. An optical processing unit performs large-scale random matrix multiplications, which is the central operation of this algorithm, at speeds up to 1500 TeraOPS under 30 Watts of power. We perform optical training of modern deep learning architectures, including Transformers, with more than 1B parameters, and obtain good performances on language, vision, and diffusion-based generative tasks. We study the scaling of the training time, and demonstrate a potential advantage of our hybrid opto-electronic approach for ultra-deep and wide neural networks, thus opening a promising route to sustain the exponential growth of modern artificial intelligence beyond traditional von Neumann approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
