基于多巴胺调制STDP的神经控制机器人强化学习
Reinforcement Learning in a Neurally Controlled Robot Using Dopamine Modulated STDP
摘要：
最近的研究表明，多巴胺调节的STDP可以解决许多与强化学习相关的问题，例如远端奖励问题。脉冲神经网络提供了一种在具体环境中实现强化学习的有用技术，因为它们可以处理连续的参数空间，并且因此更好地概括在给定环境中执行的正确行为。在这个项目中，我们实现了一个版本的DA调制STDP在一个具体的机器人上的食物觅食任务。通过模拟多巴胺能神经元，我们展示了机器人如何能够学习一系列行为以获得食物奖励。在测试中，机器人能够学习食物吸引行为，随后在所有50次试验中，当环境发生变化时忘记这种行为。此外，我们表明，机器人是能够操作的环境中，最佳的行为变化迅速，所以代理必须不断重新学习。在由食物容器组成的更复杂的环境中，机器人能够在95%的试验中学习食物容器的吸引力，尽管正确的行为和奖励之间存在很大的时间距离。这是通过将多巴胺反应从主要刺激（食物）转移到次要刺激（食物容器）来实现的。我们的工作为一些观察到的生物现象背后的原因提供了见解，例如在多巴胺能神经元中观察到的爆发行为。以及演示脉冲神经网络控制的机器人如何能够解决一系列强化学习任务。
英文摘要：
Recent work has shown that dopamine-modulated STDP can solve many of the issues associated with reinforcement learning, such as the distal reward problem. Spiking neural networks provide a useful technique in implementing reinforcement learning in an embodied context as they can deal with continuous parameter spaces and as such are better at generalizing the correct behaviour to perform in a given context. In this project we implement a version of DA-modulated STDP in an embodied robot on a food foraging task. Through simulated dopaminergic neurons we show how the robot is able to learn a sequence of behaviours in order to achieve a food reward. In tests the robot was able to learn food-attraction behaviour, and subsequently unlearn this behaviour when the environment changed, in all 50 trials. Moreover we show that the robot is able to operate in an environment whereby the optimal behaviour changes rapidly and so the agent must constantly relearn. In a more complex environment, consisting of food-containers, the robot was able to learn food-container attraction in 95% of trials, despite the large temporal distance between the correct behaviour and the reward. This is achieved by shifting the dopamine response from the primary stimulus (food) to the secondary stimulus (food-container). Our work provides insights into the reasons behind some observed biological phenomena, such as the bursting behaviour observed in dopaminergic neurons. As well as demonstrating how spiking neural network controlled robots are able to solve a range of reinforcement learning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
