结构化信用分配与协调探索
Structural Credit Assignment with Coordinated Exploration
AI总结：
本研究提出利用玻尔兹曼机或循环网络实现协调探索，以改进结构化信用分配，实验证明其训练速度优于独立探索和STE反向传播。
摘要：
一种训练人工神经网络（ANN）的生物学合理方法是将每个单元视为一个随机强化学习（RL）智能体，从而将网络视为一个智能体团队。因此，所有单元都可以通过REINFORCE进行学习，这是一种由全局奖励信号调节的局部学习规则，这与生物学上观察到的突触可塑性形式更为一致。然而，这种学习方法往往较慢，并且不能随网络规模的增大而良好扩展。这种低效率源于两个阻碍有效结构化信用分配的因素：（i）所有单元独立地探索网络，（ii）使用单一奖励来评估所有单元的动作。因此，旨在改进结构化信用分配的方法通常可分为两类。第一类包括能够在单元间实现协调探索的算法，如MAP传播。第二类包括为网络内每个单元计算更具体奖励信号的算法，如权重最大化及其变体。在本研究报告中，我们关注第一类。我们提出使用玻尔兹曼机或循环网络进行协调探索。我们表明，通常训练玻尔兹曼机所需的负相位可以被移除。由此产生的学习规则类似于奖励调节的赫布学习规则。实验结果表明，在基于REINFORCE的多个随机和离散单元的训练速度上，协调探索显著超过独立探索，甚至超越了直通估计器（STE）反向传播。
英文摘要：
A biologically plausible method for training an Artificial Neural Network (ANN) involves treating each unit as a stochastic Reinforcement Learning (RL) agent, thereby considering the network as a team of agents. Consequently, all units can learn via REINFORCE, a local learning rule modulated by a global reward signal, which aligns more closely with biologically observed forms of synaptic plasticity. However, this learning method tends to be slow and does not scale well with the size of the network. This inefficiency arises from two factors impeding effective structural credit assignment: (i) all units independently explore the network, and (ii) a single reward is used to evaluate the actions of all units. Accordingly, methods aimed at improving structural credit assignment can generally be classified into two categories. The first category includes algorithms that enable coordinated exploration among units, such as MAP propagation. The second category encompasses algorithms that compute a more specific reward signal for each unit within the network, like Weight Maximization and its variants. In this research report, our focus is on the first category. We propose the use of Boltzmann machines or a recurrent network for coordinated exploration. We show that the negative phase, which is typically necessary to train Boltzmann machines, can be removed. The resulting learning rules are similar to the reward-modulated Hebbian learning rule. Experimental results demonstrate that coordinated exploration significantly exceeds independent exploration in training speed for multiple stochastic and discrete units based on REINFORCE, even surpassing straight-through estimator (STE) backpropagation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
