Reinforcement learning (RL) problems where the learner attempts to infer an unobserved reward from some feedback variables have been studied in several recent papers. The setting of Interaction-Grounded Learning (IGL) is an example of such feedback-based RL tasks where the learner optimizes the return by inferring latent binary rewards from the interaction with the environment. In the IGL setting, a relevant assumption used in the RL literature is that the feedback variable $Y$ is conditionally independent of the context-action $(X,A)$ given the latent reward $R$. In this work, we propose Variational Information-based IGL (VI-IGL) as an information-theoretic method to enforce the conditional independence assumption in the IGL-based RL problem. The VI-IGL framework learns a reward decoder using an information-based objective based on the conditional mutual information (MI) between $(X,A)$ and $Y$. To estimate and optimize the information-based terms for the continuous random variables in the RL problem, VI-IGL leverages the variational representation of mutual information to obtain a min-max optimization problem. Also, we extend the VI-IGL framework to general $f$-Information measures leading to the generalized $f$-VI-IGL framework for the IGL-based RL problems. We present numerical results on several reinforcement learning settings indicating an improved performance compared to the existing IGL-based RL algorithm.
Imagining Grounded Conceptual Representations from Perceptual Information in Situated Guessing Games
Heriot-Watt University(赫瑞-瓦特大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Carnegie Mellon University(卡内基梅隆大学)
在视觉猜谜游戏中,猜谜者必须通过向预言者提问来识别场景中的目标对象。对于玩家来说,一个有效的策略是学习物体的概念表征,这些概念表征既有区分力又有足够的表达力,可以提出问题并正确猜测。然而,如Suglia et al.(2020)所示,现有模型无法学习真正的多模态表示,而是在训练和推理时依赖于场景中对象的黄金类别标签。当推理时的类别与训练时的类别相匹配时,这提供了不自然的性能优势,并且它导致模型在涉及域外对象类别的更真实的“zero-shot”场景中失败。为了克服这个问题,我们引入了一个基于正则化自动编码器的新型“想象”模块,该模块在推理时学习上下文感知和类别感知的潜在嵌入,而不依赖于类别标签。我们的想象力模块在CompGuessWhat?!中的游戏准确率超过了最先进的竞争对手8.26%。zero-shot方案(Suglia等人,2020),在GuessWhat?!中,它将Oracle和Guesser的准确率提高了2.08%和12.86%。基准,当推理时没有可用的黄金类别时。想象模块还可以增强对对象属性和属性的推理。