从迭代近似逆推导差分目标传播
Deriving Differential Target Propagation from Iterating Approximate Inverses
摘要:
我们表明,一种特定形式的目标传播,即,依赖于每一层的学习逆,这是差分的,即,其中目标是前向传播的小扰动,产生对应于近似的基于高斯-牛顿梯度的优化的更新规则,而不需要大矩阵的操作或求逆。有趣的是,这在生物学上比反向传播更合理,但可能会隐含地提供更强的优化过程。扩展差分目标传播,我们考虑基于每层的局部自编码器的几个迭代计算,以便实现更精确的反演以获得更准确的目标传播,并且我们表明,如果自编码函数减去恒等函数具有小于1的Lipschitz常数,则这些迭代过程以指数方式快速收敛,即,自动编码器粗略地成功执行反转。我们还提出了一种方法来规范化每一层的变化,以考虑每一层对输出的相对影响,以便在更有影响力的层上进行更大的权重变化,就像在普通的反向传播中使用梯度下降一样。
英文摘要:
We show that a particular form of target propagation, i.e., relying on learned inverses of each layer, which is differential, i.e., where the target is a small perturbation of the forward propagation, gives rise to an update rule which corresponds to an approximate Gauss-Newton gradient-based optimization, without requiring the manipulation or inversion of large matrices. What is interesting is that this is more biologically plausible than back-propagation yet may turn out to implicitly provide a stronger optimization procedure. Extending difference target propagation, we consider several iterative calculations based on local auto-encoders at each layer in order to achieve more precise inversions for more accurate target propagation and we show that these iterative procedures converge exponentially fast if the auto-encoding function minus the identity function has a Lipschitz constant smaller than one, i.e., the auto-encoder is coarsely succeeding at performing an inversion. We also propose a way to normalize the changes at each layer to take into account the relative influence of each layer on the output, so that larger weight changes are done on more influential layers, like would happen in ordinary back-propagation with gradient descent.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。