训练具有稀疏延迟奖励的循环神经网络用于灵活决策任务
Training recurrent neural networks with sparse, delayed rewards for flexible decision tasks
摘要:
混沌状态下的循环神经网络表现出复杂的动力学,让人想起行为任务期间的高水平皮层活动。然而,这种网络的现有训练方法要么在生物学上是不可信的,要么需要实时连续的误差信号来指导学习过程。这与大多数行为任务相反,行为任务只提供时间稀疏,延迟的奖励。在这里,我们表明,一个生物合理的奖励调制的赫布学习算法,以前用于前馈模型的鸟鸣学习,可以训练经常性的网络仅基于延迟,相位奖励信号在每次试验结束。该方法不需要专门的反馈或读出网络:整个网络的连通性是学习的,网络输出是从一个任意选择的网络单元读取的。我们使用这种方法成功地训练一个网络上的延迟不匹配样本任务(这需要记忆,灵活的协会,和非线性混合选择性)。使用解码技术,我们表明,由此产生的网络表现出动态编码的任务相关的信息,与神经编码的各种任务功能波动很大的过程中的审判。此外,网络活动在响应时间内从刺激特定的表征转移到响应特定的表征,这与行为动物在类似任务中的神经记录一致。我们的结论是,循环神经网络,奖励调制的赫布学习训练,提供了一个合理的模型,在学习和灵活的协会性能的皮质动力学。
英文摘要:
Recurrent neural networks in the chaotic regime exhibit complex dynamics reminiscent of high-level cortical activity during behavioral tasks. However, existing training methods for such networks are either biologically implausible, or require a real-time continuous error signal to guide the learning process. This is in contrast with most behavioral tasks, which only provide time-sparse, delayed rewards. Here we show that a biologically plausible reward-modulated Hebbian learning algorithm, previously used in feedforward models of birdsong learning, can train recurrent networks based solely on delayed, phasic reward signals at the end of each trial. The method requires no dedicated feedback or readout networks: the whole network connectivity is subject to learning, and the network output is read from one arbitrarily chosen network cell. We use this method to successfully train a network on a delayed nonmatch to sample task (which requires memory, flexible associations, and non-linear mixed selectivities). Using decoding techniques, we show that the resulting networks exhibit dynamic coding of task-relevant information, with neural encodings of various task features fluctuating widely over the course of a trial. Furthermore, network activity moves from a stimulus-specific representation to a response-specific representation during response time, in accordance with neural recordings in behaving animals for similar tasks. We conclude that recurrent neural networks, trained with reward-modulated Hebbian learning, offer a plausible model of cortical dynamics during learning and performance of flexible association.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。