arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.04736cs.CLcs.AI
收藏

反馈就是你所需的一切吗?在目标条件强化学习中利用自然语言反馈

Is Feedback All You Need? Leveraging Natural Language Feedback in Goal-Conditioned Reinforcement Learning

University of Edinburgh(爱丁堡大学);Heriot-Watt University(赫瑞-瓦特大学)

Sabrina McCallum, Max Taylor-Davies, Stefano V. Albrecht, Alessandro Suglia

AI总结:

本文扩展 BabyAI 自动生成自然语言反馈,并改造 Decision Transformer 利用该信号,证明其可替代或补充回报与目标描述,提升目标条件 RL 智能体的泛化能力。

摘要:

尽管取得了众多成功,强化学习(RL)领域仍远未达到人类行为学习所展现的出色泛化能力。帮助弥合这一差距的一种可能方法,是向 RL 智能体提供以自然语言表达的、更丰富且更接近人类的反馈。为研究这一想法,我们首先扩展 BabyAI,使其能够根据环境动态和目标条件成功情况自动生成语言反馈。随后,我们修改 Decision Transformer 架构,以利用这一额外信号。我们发现,使用语言反馈替代 return-to-go 或目标描述,或将其作为二者的补充进行训练,都能提升智能体的泛化性能;而且,即使反馈仅在训练期间可用、在推理时不可用,智能体仍可从中受益。

英文摘要:

Despite numerous successes, the field of reinforcement learning (RL) remains far from matching the impressive generalisation power of human behaviour learning. One possible way to help bridge this gap be to provide RL agents with richer, more human-like feedback expressed in natural language. To investigate this idea, we first extend BabyAI to automatically generate language feedback from the environment dynamics and goal condition success. Then, we modify the Decision Transformer architecture to take advantage of this additional signal. We find that training with language feedback either in place of or in addition to the return-to-go or goal descriptions improves agents' generalisation performance, and that agents can benefit from feedback even when this is only available during training, but not at inference.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑