arXiv 2604.030372026-04-21cs.ROcs.AIcs.CV

ARM：用于长 horizon 操控的优势奖励建模

ARM: Advantage Reward Modeling for Long-Horizon Manipulation

LimX Dynamics（LimX动力学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Zhejiang University（浙江大学）

Yiming Mao, Zixi Yu, Weixin Mao, Yinhao Li, Qirui Hu, Zihan Lan, Minzhao Zhu, Hua Chen

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出ARM框架，通过三态标签策略提升长 horizon 操控任务的奖励建模，实现高效稳定的数据利用。

摘要：

长 horizon 机器人操控仍使强化学习面临挑战，因稀疏奖励对信用分配指导有限。实际策略改进依赖更丰富的中间监督，如密集进度奖励，但获取成本高且不适合非单调行为。为此，我们提出优势奖励建模（ARM），从难以量化绝对进步转向估计相对优势。引入成本有效的三态标签策略——渐进、退化和停滞，减少人工认知负担并确保高跨标注者一致性。通过训练这些直观信号，ARM可自动标注完整演示和碎片化DAgger数据。将ARM整合到离线RL流程中，实现自适应动作-奖励重加权，有效过滤次优样本。我们的方法在具有挑战性的长 horizon 毛巾折叠任务上实现了99.4%的成功率，证明了在接近零人工干预下，相较于当前VLA基线，具有改进的稳定性与数据效率。

英文摘要：

Long-horizon robotic manipulation remains challenging for reinforcement learning (RL) because sparse rewards provide limited guidance for credit assignment. Practical policy improvement thus relies on richer intermediate supervision, such as dense progress rewards, which are costly to obtain and ill-suited to non-monotonic behaviors such as backtracking and recovery. To address this, we propose Advantage Reward Modeling (ARM), a framework that shifts from hard-to-quantify absolute progress to estimating relative advantage. We introduce a cost-effective tri-state labeling strategy -- Progressive, Regressive, and Stagnant -- that reduces human cognitive overhead while ensuring high cross-annotator consistency. By training on these intuitive signals, ARM enables automated progress annotation for both complete demonstrations and fragmented DAgger-style data. Integrating ARM into an offline RL pipeline allows for adaptive action-reward reweighting, effectively filtering suboptimal samples. Our approach achieves a 99.4% success rate on a challenging long-horizon towel-folding task, demonstrating improved stability and data efficiency over current VLA baselines with near-zero human intervention during policy training.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文