用于连续动作片段的Actor-Critic：一种针对长周期机械臂操作的强化学习框架
Actor-Critic for Continuous Action Chunks: A Reinforcement Learning Framework for Long-Horizon Robotic Manipulation with Sparse Reward
AI总结：
AC3是一种用于连续动作分块的强化学习框架，通过定向稳定机制和自监督模块，在稀疏奖励环境下实现高效稳定的动作序列学习。
摘要：
现有的强化学习（RL）方法在处理长周期机械臂操作任务时面临挑战，特别是涉及稀疏奖励的任务。尽管动作分块是机器人操作中的一种有前途的范式，但使用RL直接学习连续动作分块以稳定且高效的方式仍然是一项关键挑战。本文介绍了一个名为AC3（用于连续分块的Actor-Critic）的新RL框架，该框架能够学习生成高维连续动作序列。为了使这一学习过程稳定且高效，AC3集成了针对演员和评论家的定向稳定机制。首先，为了确保可靠的策略改进，演员使用不对称更新规则进行训练，仅从成功的轨迹中学习。其次，为了在稀疏奖励下实现有效的价值学习，评论家的更新通过内分块n步回报进行稳定，并通过自监督模块在每个动作分块的锚点处提供内在奖励。我们在BiGym和RLBench基准上的25个任务上进行了广泛的实验。结果表明，通过仅使用少量演示和简单的模型架构，AC3在大多数任务上实现了更高的成功率，验证了其有效设计。
英文摘要：
Existing reinforcement learning (RL) methods struggle with long-horizon robotic manipulation tasks, particularly those involving sparse rewards. While action chunking is a promising paradigm for robotic manipulation, using RL to directly learn continuous action chunks in a stable and data-efficient manner remains a critical challenge. This paper introduces AC3 (Actor-Critic for Continuous Chunks), a novel RL framework that learns to generate high-dimensional, continuous action sequences. To make this learning process stable and data-efficient, AC3 incorporates targeted stabilization mechanisms for both the actor and the critic. First, to ensure reliable policy improvement, the actor is trained with an asymmetric update rule, learning exclusively from successful trajectories. Second, to enable effective value learning despite sparse rewards, the critic's update is stabilized using intra-chunk $n$-step returns and further enriched by a self-supervised module providing intrinsic rewards at anchor points aligned with each action chunk. We conducted extensive experiments on 25 tasks from the BiGym and RLBench benchmarks. Results show that by using only a few demonstrations and a simple model architecture, AC3 achieves superior success rates on most tasks, validating its effective design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
