arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1902.07198cs.LGcs.AIcs.CLstat.ML
收藏

学习从稀疏和未指定的奖励中概括

Learning to Generalize from Sparse and Underspecified Rewards

Google Research(谷歌研究院);University of Alberta(阿尔伯塔大学)

Rishabh Agarwal, Chen Liang, Dale Schuurmans, Mohammad Norouzi

摘要:

我们考虑从稀疏和未指定的奖励,其中代理接收复杂的输入,如自然语言指令,并需要生成一个复杂的响应,如动作序列,而只接收二进制成功失败的反馈学习的问题。这种成功-失败的奖励往往是不明确的:它们不区分有目的的成功和偶然的成功。从未指定的奖励中进行推广取决于对获得意外成功的虚假轨迹的贴现,而从稀疏反馈中学习则需要有效的探索。我们通过使用覆盖KL发散方向的模式来收集一组不同的成功轨迹,然后通过寻求KL发散的模式来训练鲁棒的策略来解决探索问题。我们提出了Meta Reward Learning(MeRL)来构造一个辅助的奖励函数,为学习提供更精细的反馈。辅助奖励函数的参数相对于训练策略的验证性能进行优化。MeRL方法优于我们基于贝叶斯优化的替代奖励学习技术,并在弱监督语义解析方面达到了最先进的水平。在WikiTableQuestions和WikiSQL数据集上,它分别提高了1.2%和2.4%。

英文摘要:

We consider the problem of learning from sparse and underspecified rewards, where an agent receives a complex input, such as a natural language instruction, and needs to generate a complex response, such as an action sequence, while only receiving binary success-failure feedback. Such success-failure rewards are often underspecified: they do not distinguish between purposeful and accidental success. Generalization from underspecified rewards hinges on discounting spurious trajectories that attain accidental success, while learning from sparse feedback requires effective exploration. We address exploration by using a mode covering direction of KL divergence to collect a diverse set of successful trajectories, followed by a mode seeking KL divergence to train a robust policy. We propose Meta Reward Learning (MeRL) to construct an auxiliary reward function that provides more refined feedback for learning. The parameters of the auxiliary reward function are optimized with respect to the validation performance of a trained policy. The MeRL approach outperforms our alternative reward learning technique based on Bayesian Optimization, and achieves the state-of-the-art on weakly-supervised semantic parsing. It improves previous work by 1.2% and 2.4% on WikiTableQuestions and WikiSQL datasets respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑