备忘录：通过强化学习训练内存高效的具身智能体
Memo: Training Memory-Efficient Embodied Agents with Reinforcement Learning
AI总结：
Memo通过在训练过程中交错周期性总结标记与输入，实现内存高效的具身智能体强化学习训练，优于长上下文基线并更高效。
摘要：
为了使具身智能体在长时间内有效运作，开发能够形成和访问记忆的模型至关重要，以保持其在环境中的上下文性。在当前训练基于变压器的策略以处理具身序列决策任务的范式中，视觉输入常常超出变压器的上下文限制，而人类可以将一生的经验压缩为记忆来维持和利用。理论上可以实现显著压缩，因为大部分输入是无关的，可以被抽象化。然而，现有方法大多专注于具有固定大小记忆的递归模型或完全依赖全上下文的变压器。在本工作中，我们提出了Memo，一种基于变压器的架构和强化学习（RL）的训练方法，用于处理内存密集型、长时间跨度的任务。Memo通过在训练过程中将周期性总结标记与模型输入交错，从而实现记忆的创建和检索。我们在网格世界元RL基准和照片现实感室内的多物体导航任务中展示了Memo的有效性。Memo在性能上优于朴素的长上下文变压器基线，同时在计算和存储效率上更具优势。此外，Memo在推理时间能更好地泛化到更长的上下文，并且在流式设置中保持稳健，其中历史上下文必须截断以适应推理约束。我们的代码可在：https://github.com/gunshi/memo获取。
英文摘要：
To enable embodied agents to operate effectively over extended timeframes, it is crucial to develop models that form and access memories to stay contextualized in their environment. In the current paradigm of training transformer-based policies for embodied sequential decision-making tasks, visual inputs often overwhelm the context limits of transformers, while humans can maintain and utilize a lifetime of experience compressed as memories. Significant compression is possible in principle, as much of the input is irrelevant and can be abstracted. However, existing approaches predominantly focus on either recurrent models with fixed-size memory or transformers with full-context reliance. In this work, we propose Memo, a transformer-based architecture and training recipe for reinforcement learning (RL) on memory-intensive, long-horizon tasks. Memo incorporates the creation and retrieval of memory by interleaving periodic summarization tokens with the inputs of a model during training. We demonstrate Memo's effectiveness on a gridworld meta-RL benchmark and a multi-object navigation task in photo-realistic indoor settings. Memo outperforms naive long-context transformer baselines while being more compute and storage efficient. Additionally, Memo generalizes better to longer contexts at inference time and remains robust in streaming settings, where historical context must be truncated to fit inference constraints. Our code is available at: https://github.com/gunshi/memo.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
