arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.00719cs.RO
收藏

关键帧引导的结构化奖励用于长周期实验室机器人强化学习

Keyframe-Guided Structured Rewards for Reinforcement Learning in Long-Horizon Laboratory Robotics

Suzhou Institute for Advanced Research, University of Science and Technology of China(科学技术大学苏州市先进研究院)

Yibo Qiu, Shu'ang Sun, Haoliang Ye, Ronald X Xu, Mingzhai Sun

AI总结:

本文提出关键帧引导的结构化奖励方法,用于提升实验室机器人在长周期任务中的精确操作能力。

摘要:

在实验室自动化中,如移液头附件和液体转移的长周期精确操作,需要在连续、高维状态空间中运行的策略,以尊重严格的程序逻辑。然而,现有方法在奖励稀疏性、多阶段结构约束和噪声或不完美的演示方面存在困难,导致探索效率低下和收敛不稳定。我们提出了一种关键帧引导的奖励生成框架,该框架能够自动从演示中提取运动学感知的关键帧,通过基于扩散的预测器在潜在空间中生成阶段性目标,并构建基于几何进展的奖励以引导在线强化学习。该框架集成了多视角视觉编码、基于潜在相似性的进展跟踪和在视觉-语言-动作骨干上的人类在环强化学习微调,以使策略优化与生物协议的内在逐步逻辑对齐。在四个现实世界实验室任务中,包括高精度移液头附件和动态液体转移,我们的方法在40-60分钟的在线微调后,平均成功率为82%。与HG-DAgger(42%)和Hil-ConRFT(47%)相比,我们的方法展示了结构化关键帧引导奖励在克服探索瓶颈和提供高精度、长周期机器人实验室自动化可扩展解决方案方面的有效性。

英文摘要:

Long-horizon precision manipulation in laboratory automation, such as pipette tip attachment and liquid transfer, requires policies that respect strict procedural logic while operating in continuous, high-dimensional state spaces. However, existing approaches struggle with reward sparsity, multi-stage structural constraints, and noisy or imperfect demonstrations, leading to inefficient exploration and unstable convergence. We propose a Keyframe-Guided Reward Generation Framework that automatically extracts kinematics-aware keyframes from demonstrations, generates stage-wise targets via a diffusion-based predictor in latent space, and constructs a geometric progress-based reward to guide online reinforcement learning. The framework integrates multi-view visual encoding, latent similarity-based progress tracking, and human-in-the-loop reinforcement fine-tuning on a Vision-Language-Action backbone to align policy optimization with the intrinsic stepwise logic of biological protocols. Across four real-world laboratory tasks, including high-precision pipette attachment and dynamic liquid transfer, our method achieves an average success rate of 82% after 40--60 minutes of online fine-tuning. Compared with HG-DAgger (42%) and Hil-ConRFT (47%), our approach demonstrates the effectiveness of structured keyframe-guided rewards in overcoming exploration bottlenecks and providing a scalable solution for high-precision, long-horizon robotic laboratory automation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑