arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.05015cs.LG
收藏

一种面向交互接地学习的信息论方法

An Information Theoretic Approach to Interaction-Grounded Learning

The Chinese University of Hong Kong(香港中文大学)

Xiaoyan Hu, Farzan Farnia, Ho-fung Leung

AI总结:

针对交互接地学习中反馈变量与上下文-动作对的条件独立假设落地问题,提出基于变分互信息的VI-IGL框架及广义f-VI-IGL框架,通过极小-极大优化学习奖励解码器,实验证明其性能优于现有同类强化学习算法。

摘要:

近期多篇论文研究了强化学习(RL)中智能体尝试从若干反馈变量推断未观测奖励的问题。交互接地学习(Interaction-Grounded Learning,IGL)场景正是这类基于反馈的强化学习任务的典型代表,其中智能体通过推断与环境交互过程中的潜在二元奖励来优化回报。在IGL场景下,强化学习文献中常用的一个相关假设是:在给定潜在奖励$R$的条件下,反馈变量$Y$与上下文-动作对$(X,A)$条件独立。本研究提出基于变分信息的交互接地学习(Variational Information-based IGL,VI-IGL),作为一种信息论方法,在基于IGL的强化学习问题中强制执行上述条件独立假设。VI-IGL框架利用基于$(X,A)$与$Y$之间条件互信息(MI)的信息论目标函数,学习一个奖励解码器。为估计并优化强化学习问题中连续随机变量对应的信息论项,VI-IGL借助互信息的变分表示,将其转化为一个极小-极大优化问题。此外,我们将VI-IGL框架扩展至通用的$f$-信息测度,得到适用于基于IGL的强化学习问题的广义$f$-VI-IGL框架。我们在多个强化学习场景下开展了数值实验,结果表明该方法相比现有基于IGL的强化学习算法性能更优。

英文摘要:

Reinforcement learning (RL) problems where the learner attempts to infer an unobserved reward from some feedback variables have been studied in several recent papers. The setting of Interaction-Grounded Learning (IGL) is an example of such feedback-based RL tasks where the learner optimizes the return by inferring latent binary rewards from the interaction with the environment. In the IGL setting, a relevant assumption used in the RL literature is that the feedback variable $Y$ is conditionally independent of the context-action $(X,A)$ given the latent reward $R$. In this work, we propose Variational Information-based IGL (VI-IGL) as an information-theoretic method to enforce the conditional independence assumption in the IGL-based RL problem. The VI-IGL framework learns a reward decoder using an information-based objective based on the conditional mutual information (MI) between $(X,A)$ and $Y$. To estimate and optimize the information-based terms for the continuous random variables in the RL problem, VI-IGL leverages the variational representation of mutual information to obtain a min-max optimization problem. Also, we extend the VI-IGL framework to general $f$-Information measures leading to the generalized $f$-VI-IGL framework for the IGL-based RL problems. We present numerical results on several reinforcement learning settings indicating an improved performance compared to the existing IGL-based RL algorithm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑