arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.12571cs.LGcond-mat.soft
收藏

一种全晶体管物理学习Transformer可被训练以执行上下文学习

An all-transistor physical-learning transformer can be trained to perform in-context learning

Yale Physics & QBio(耶鲁大学物理与QBIO)

Michael C. Abbott, Benjamin B. Machta

AI总结:

本研究提出一种全晶体管物理学习Transformer,通过引入额外电路元件恢复双向流动,构建基于亚阈值MOSFET的注意力机制,经Lion优化器训练可执行上下文学习任务,需交叉熵损失及新型浮置电路元件实现训练。

摘要:

大多数通过平衡传播或耦合学习训练的系统属于与多层感知机相同的类别:虽然它们可能是通用函数逼近器,但难以学习关联任务,而关联任务对Transformer来说很容易。平衡传播依赖于信息通过输入和输出之间的物理系统的双向流动,而反向流动会被更复杂的非线性系统破坏,例如表达点积注意力所需的乘法运算。在此,我们通过在双电路或二元电路中引入额外的电路元件,明确恢复了这种双向流动。随后,我们利用这些思路构建了一种注意力机制,该机制利用亚阈值MOSFET器件的指数电流定律。我们在模拟中表明,尽管使用仅传输近似梯度的真实反向元件,该电路仍可被训练以执行简单的上下文学习任务;为使这种训练成功,我们需要交叉熵损失(而非均方误差),并使用基于softmax所用相同MOSFET子电路的新型浮置电路元件生成适合该损失的微调。我们使用Lion优化器训练该电路中的电阻,该优化器可通过每个参数使用一个电容来存储过去梯度估计的局部记忆。

英文摘要:

Most systems trained by equilibrium propagation or coupled learning are in the same class as multi-layer perceptrons: while they may be universal function approximators, they struggle to learn associative tasks which are easy for transformers. Equilibrium propagation relies on a bidirectional flow of information through the physical system between input and output. The reverse flow is broken by more complex nonlinear systems, such as those performing multiplication which is needed to express dot-product attention. Here we explicitly restore this bidirectional flow by introducing extra circuit elements for this purpose into a double or dyadic circuit. We then use these ideas to construct an attention mechanism exploiting the exponential current law of sub-threshold MOSFET devices. We show that in simulations this circuit can be trained to perform a simple in-context learning task, despite using realistic reverse components which transmit only an approximate gradient. For this training to succeed we need cross-entropy loss (not mean squared error). We generate nudges appropriate for this loss using a novel floating circuit element, based on the same MOSFET sub-circuit used for softmax. We train the resistors in this circuit using the Lion optimiser, which can be implemented using one capacitor per parameter to store local memory of past gradient estimates.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑