arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2105.11233cs.NEcs.ETcs.LG
收藏

通过零差梯度提取实现材料内部梯度下降

Gradient descent in materia through homodyne gradient extraction

University of Twente(特温特大学);Technische Universität Berlin(柏林工业大学);Eindhoven University of Technology(埃因霍温理工大学);University of Münster(明斯特大学)

Marcus N. Boon, Lorenzo Cassola, Hans-Christian Ruiz Euler, Tao Chen, Bram van de Ven, Unai Alegre Ibarra, Peter A. Bobbert, Wilfred G. van der Wiel

AI总结:

本文提出基于零差检测的梯度提取方法,通过不同频率正弦扰动参数,直接在物理系统中获得损失函数梯度并实现梯度下降,且在掺杂网络处理单元中验证。

摘要:

深度学习是一种受大脑启发的多层神经网络方法,已经彻底改变了机器学习。其关键推动因素之一是反向传播,这是一种计算损失函数相对于神经网络模型中权重和偏置的梯度的算法,并与梯度下降结合使用。然而,在数字计算机上实现深度学习本质上十分耗能,对许多应用而言,能耗已高到令人难以承受。这推动了专用硬件的发展,涵盖神经形态 CMOS 集成电路、集成光子张量核心,以及非传统的、基于材料的计算系统。这些材料系统中的学习过程,例如通过人工进化、平衡传播或代理建模实现,是一个复杂且耗时的过程。在这里,我们展示了一种简单但高效且准确的梯度提取方法,该方法基于零差检测原理,用于直接在物理系统中对损失函数执行梯度下降,而无需解析描述。通过使用具有不同频率的正弦波形扰动需要优化的参数,我们以高度鲁棒且可扩展的方式有效获得梯度信息。我们在掺杂网络处理单元中演示了该方法,但认为它适用于广泛的物理系统。零差梯度提取原则上可以完全在材料内部实现,从而促进自主学习材料系统的发展。

英文摘要:

Deep learning, a multi-layered neural network approach inspired by the brain, has revolutionized machine learning. One of its key enablers has been backpropagation, an algorithm that computes the gradient of a loss function with respect to the weights and biases in the neural network model, in combination with its use in gradient descent. However, the implementation of deep learning in digital computers is intrinsically energy hungry, with energy consumption becoming prohibitively high for many applications. This has stimulated the development of specialized hardware, ranging from neuromorphic CMOS integrated circuits and integrated photonic tensor cores to unconventional, material-based computing system. The learning process in these material systems, realized, e.g., by artificial evolution, equilibrium propagation or surrogate modelling, is a complicated and time-consuming process. Here, we demonstrate a simple yet efficient and accurate gradient extraction method, based on the principle of homodyne detection, for performing gradient descent on a loss function directly in a physical system without the need of an analytical description. By perturbing the parameters that need to be optimized using sinusoidal waveforms with distinct frequencies, we effectively obtain the gradient information in a highly robust and scalable manner. We illustrate the method in dopant network processing units, but argue that it is applicable in a wide range of physical systems. Homodyne gradient extraction can in principle be fully implemented in materia, facilitating the development of autonomously learning material systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑