ETHER：为事后经验回放对齐涌现通信
ETHER: Aligning Emergent Communication for Hindsight Experience Replay
AI总结：
本文提出ETHER，利用涌现通信解决指令跟随任务中HER的目标重标记与谓词函数缺失问题，通过指称游戏训练说话者和听者并部分对齐语言，在BabyAI任务上提升样本效率。
摘要：
事后经验回放（HER）通过用实际实现的目标重新标记失败的轨迹，提高了目标条件强化学习（RL）中的样本效率。然而，HER假设可以访问目标重新标记函数和用于确定目标是否已满足的谓词函数。这些假设在指令跟随任务中不成立，因为此类任务的目标以自然语言表达，且与状态空间不同。我们将此形式化为事后强化学习问题，表明需要将RL策略与这些函数联合学习。为解决此问题，我们提出ETHER（涌现文本事后经验回放），一种利用涌现通信的智能体。ETHER使用指称游戏（RG）训练说话者和听者，以开发一种描述环境状态的、基于实体的人工语言。它利用任务指令与RL观测之间的共现模式，将这种涌现语言与指令语言部分对齐。在BabyAI的PickupDist任务上的实验表明，ETHER学习到的RG说话者和听者可以作为HER的目标重新标记和谓词函数，尽管语言对齐不完美，仍能提高样本效率。我们的工作弥合了涌现通信与目标条件RL之间的鸿沟，为HER的更广泛应用打开了大门。
英文摘要：
Hindsight Experience Replay (HER) enhances sample efficiency in goal-conditioned reinforcement learning (RL) by relabelling failed trajectories with goals that were actually achieved. However, HER implicitly assumes access to a goal relabelling function and a predicate function that determines whether a goal has been satisfied. These assumptions break down in instruction-following tasks, where goals are expressed in natural language and differ from the state space. We formalize this as the Hindsight Reinforcement Learning problem, which shows the need to jointly learn these functions alongside the RL policy. To address it, we propose ETHER (Emergent Textual Hindsight Experience Replay), an agent that leverages Emergent Communication to learn the goal-relabelling and predicate functions. ETHER uses a referential game (RG) to train a speaker and a listener to develop a grounded, artificial language describing environment states. It partially aligns this emergent language with instruction language using co-occurrence patterns between task instructions and RL observations. We prove that the relabelling and predicate functions that ETHER derives from the RG avoid the degenerate solutions of the Hindsight RL problem, namely trivial predicates and collapsed relabelling functions. Experiments on BabyAI's PickupDist task show that ETHER's learned RG speaker and listener can function as the goal relabelling and predicate functions of HER, improving sample efficiency despite imperfect language alignment. Our work bridges Emergent Communication and goal-conditioned RL, opening the door to wider applications of HER.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
