arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.14409cs.LGcs.ETcs.NE
收藏

PotentialEP:基于平衡传播的无反向传播训练CIFAR规模生成模型

PotentialEP: Backprop-Free Training of CIFAR-Scale Generative Models via Equilibrium Propagation

University of Cambridge(剑桥大学);Nokia Bell Labs(诺基亚贝尔实验室)

Alex Gower

AI总结:

PotentialEP通过在预测编码网络能量中添加线性注入项,实现无反向传播训练CIFAR规模生成模型,其FID与原模型差距小,是首个用平衡传播训练的CIFAR-10生成模型。

摘要:

用扩散模型或流匹配模型生成图像通常需要对速度场进行数百次顺序积分,这种计算对GPU效率低,但适合模拟神经形态硬件,因为一次快速低功耗的物理松弛即可替代整个序列。这需要两个条件:一是计算过程为松弛的生成算法,二是无需反向传播的训练方法,因为反向传播的全局梯度传输难以在模拟硬件中实现。能量匹配提供了第一个条件,它通过对时间无关势的梯度下降松弛进行采样,但该下降遵循的梯度本身是每次积分步骤的反向传播。我们证明,在能量中添加单个线性注入项的预测编码网络会松弛到一个状态,其可见层固有携带该梯度,且当注入强度较小时误差会消失。因此,经反向传播训练的能量匹配模型可直接加载到该网络中,仅通过松弛采样,在CIFAR-10和ImageNet-32规模下与原模型的FID差距在0.04以内。该网络也可无反向传播训练:平衡传播重现反向传播的参数更新,余弦相似度达0.9998,用这些更新端到端训练在无条件CIFAR-10上达到FID 3.64,而反向传播为3.54。据我们所知,这是首个用平衡传播训练的CIFAR-10生成模型,也是首个通过任何无反向传播规则达到竞争力质量的模型。该质量无需组归一化或注意力:仅由卷积和逐点非线性组成的架构(两者均为模拟交叉开关的原生组件)经反向传播训练时FID达3.51。

英文摘要:

Generating an image with a diffusion or flow-matching model typically involves integrating a velocity field for hundreds of sequential steps. That computation suits a GPU poorly and an analog neuromorphic substrate well, where one fast, low-power physical relaxation could replace the whole sequence. Two things are needed: a generative algorithm whose computation is a relaxation, and a way to train it without backpropagation, whose global gradient transport is hard to realise in analog. Energy Matching supplies the first, sampling by gradient-descent relaxation on a time-independent potential - but obtaining the gradient that descent follows is itself a backward pass, once per integration step. We prove that a predictive coding network with a single linear injection term added to its energy relaxes to a state whose visible layer carries that gradient inherently, with an error that vanishes for small injection strengths. An Energy Matching model trained by backpropagation can therefore be loaded into this network unchanged and sampled by relaxation alone, matching the original model's FID to within 0.04 at CIFAR-10 and ImageNet-32 scale. The network can also be trained without backpropagation: Equilibrium Propagation reproduces backpropagation's parameter updates to a cosine similarity of 0.9998, and training end to end with those updates reaches an FID of 3.64 on unconditional CIFAR-10 against 3.54 for backpropagation. To our knowledge this is the first CIFAR-10 generative model trained with Equilibrium Propagation, and the first at competitive quality by any backpropagation-free rule. This quality does not require group normalisation or attention: an architecture of only convolutions and pointwise nonlinearities, both native to an analog crossbar, reaches an FID of 3.51 when trained by backpropagation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑