用于离线对比目标条件强化学习的反事实回滚
Counterfactual Rollouts for Offline Contrastive Goal-Conditioned Reinforcement Learning
AI总结：
该研究针对离线对比目标条件强化学习中存在的虚假动作-结果关联问题，提出CauCRL算法，通过反事实转换与回滚训练提升了混淆目标到达任务的成功率。
摘要：
对比强化学习（CRL）通过将状态-动作对与后续达到的状态关联来学习目标条件值，前提是没有未观测变量同时影响记录的动作及其结果。当离线演示来自学习者无法获取其信息的演示者时，该假设不成立，评论家会吸收虚假的动作-结果关联。我们用混淆马尔可夫决策过程形式化此设置，并引入反事实转换以区分数据可识别的内容与不可识别的内容；在所有兼容转换上取最小值可得到一个最坏情况的占用率，其为真实占用率的下界。因果对比强化学习（CauCRL）从正、负离线演示中学习悲观反事实转换，并在反事实回滚上训练评论家，在混淆目标到达任务上，其成功率显著优于标准CRL。
英文摘要：
Contrastive reinforcement learning (CRL) learns goal-conditioned values by associating state--action pairs with the states subsequently reached, assuming that no unobserved variable jointly affects the recorded action and its outcome. When offline demonstrations come from a demonstrator with information hidden from the learner, this assumption fails and the critic absorbs spurious action--outcome associations. We formalize this setting with confounded Markov decision processes and introduce counterfactual transitions that separate what the data identify from what they do not; minimizing over all compatible transitions yields a worst-case occupancy that lower-bounds the true one. Causal Contrastive Reinforcement Learning (CauCRL) learns a pessimistic counterfactual transition from positive and negative offline demonstrations and trains the critic on counterfactual rollouts, substantially improving success rates over standard CRL on confounded goal-reaching tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
