arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1704.08795cs.CL
收藏

利用强化学习将指令和视觉观察映射到动作

Mapping Instructions and Visual Observations to Actions with Reinforcement Learning

Cornell University(康奈尔大学);Microsoft Research(微软研究院)

Dipendra Misra, John Langford, Yoav Artzi

摘要:

我们建议直接将原始视觉观察和文本输入映射到指令执行的动作。虽然现有的方法假设访问结构化的环境表示或使用单独训练的模型的管道,但我们学习单个模型来联合推理语言和视觉输入。我们使用强化学习在上下文强盗设置训练一个神经网络代理。为了引导代理人的探索,我们使用奖励塑造与不同形式的监督。我们的方法不需要中间表示,规划程序或训练不同的模型。我们在模拟环境中进行了评估,并显示出对监督学习和常见强化学习变体的显着改进。

英文摘要:

We propose to directly map raw visual observations and text input to actions for instruction execution. While existing approaches assume access to structured environment representations or use a pipeline of separately trained models, we learn a single model to jointly reason about linguistic and visual input. We use reinforcement learning in a contextual bandit setting to train a neural network agent. To guide the agent's exploration, we use reward shaping with different forms of supervision. Our approach does not require intermediate representations, planning procedures, or training different models. We evaluate in a simulated environment, and show significant improvements over supervised learning and common reinforcement learning variants.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑