CONTHER：面向稀疏奖励机器人操作的情境感知强化学习
CONTHER: Context-Aware Reinforcement Learning for Robotic Manipulation with Sparse Rewards
AI总结：
本文提出CONTHER，结合Transformer架构与改进的HER回放缓冲区，利用运动历史增强稀疏奖励下的目标条件强化学习，在UR3模拟任务中显著提升成功率与稳定性。
摘要：
本文研究了在稀疏奖励操作任务中，顺序情境是否能够改善目标条件强化学习。虽然事后经验回放（HER）通过目标重标记解决了奖励稀疏性问题，但其对孤立转换的操作限制了其捕捉关节空间控制中固有时间依赖性的能力。我们假设引入运动历史可以增强策略学习，并提出了CONTHER，它将基于Transformer的架构与改进的HER回放缓冲区相结合。Transformer编码先前状态和目标序列以提供时间感知，而缓冲区则用人工成功的轨迹填充经验。分析了两种架构变体，以研究情境信息应如何整合。在使用UR3机械臂的模拟点到达任务中，CONTHER相比基线实现了38.46%更高的平均成功率，并比最强基线高出28.21%，且收敛更快、学习更稳定。该框架还在三个需要复杂轨迹跟踪和避障的动态任务上进行了评估，这些任务中时间情境至关重要。通过直接对关节速度进行操作，该方法为向物理系统迁移提供了基础。主要贡献是对顺序情境与目标重标记融合的系统性研究，为时间感知如何有益于策略学习提供了见解。
英文摘要：
This paper investigates whether sequential context improves goal-conditioned Reinforcement Learning in sparse-reward manipulation tasks. While Hindsight Experience Replay (HER) addresses reward sparsity through goal relabeling, its operation on isolated transitions limits its ability to capture temporal dependencies inherent in joint-space control. We hypothesize that incorporating motion history can enhance policy learning and introduce CONTHER, which integrates a Transformer-based architecture with a modified HER replay buffer. The Transformer encodes sequences of prior states and goals to provide temporal awareness, while the buffer populates experience with artificially successful trajectories. Two architectural variants are analyzed to examine how contextual information should be integrated. In simulated point-reaching tasks with a UR3 manipulator, CONTHER achieves a 38.46% higher average success rate compared to baselines and outperforms the strongest baseline by 28.21%, with faster convergence and more stable learning. The framework is further evaluated on three dynamic tasks requiring complex trajectory following and obstacle avoidance, where temporal context is critical. By operating directly on joint velocities, the approach provides a foundation for transfer to physical systems. The primary contribution is a systematic investigation into fusing sequential context with goal relabeling, offering insights into how temporal awareness benefits policy learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
