RoboAct-CLIP：面向机器人学的原子动作理解视频驱动预训练
RoboAct-CLIP: Video-Driven Pre-training of Atomic Action Understanding for Robotics
AI总结：
本文提出RoboAct-CLIP，通过构建单一原子动作数据集重构框架和基于CLIP的时间解耦微调策略，解决现有VLMs在机器人操作中时间动作语义建模不足的问题，使成功率提升12%并增强泛化能力。
摘要：
视觉语言模型（VLMs）已成为机器人系统的关键工具，通过多模态感知和语义推理实现跨任务泛化、动态环境交互和长程规划。然而，现有的开源VLMs主要针对通用视觉-语言对齐任务进行训练，无法有效建模对机器人操作至关重要的时间相关动作语义。尽管当前基于图像的微调方法部分地将VLMs适配于机器人应用，但它们从根本上忽视了视频序列中的时间演化模式，且在机器人智能体、被操作物体和环境上下文之间存在视觉特征纠缠，从而限制了原子动作的语义解耦能力并损害了模型泛化性。为克服这些挑战，本文提出了RoboAct-CLIP，具有双重技术贡献：1）一个数据集重构框架，对开源机器人视频执行语义约束的动作单元分割和重新标注，构建包含单一原子动作（例如“抓取”）的纯化训练集；2）一种基于对比语言-图像预训练（CLIP）架构的时间解耦微调策略，该策略将跨视频帧的时间动作特征与以对象为中心的特征解耦，以实现机器人原子动作的分层表示学习。在模拟环境中的实验结果表明，RoboAct-CLIP预训练模型的成功率比基线VLMs高12%，并在多物体操作任务中展现出卓越的泛化能力。
英文摘要：
Visual Language Models (VLMs) have emerged as pivotal tools for robotic systems, enabling cross-task generalization, dynamic environmental interaction, and long-horizon planning through multimodal perception and semantic reasoning. However, existing open-source VLMs predominantly trained for generic vision-language alignment tasks fail to model temporally correlated action semantics that are crucial for robotic manipulation effectively. While current image-based fine-tuning methods partially adapt VLMs to robotic applications, they fundamentally disregard temporal evolution patterns in video sequences and suffer from visual feature entanglement between robotic agents, manipulated objects, and environmental contexts, thereby limiting semantic decoupling capability for atomic actions and compromising model generalizability.To overcome these challenges, this work presents RoboAct-CLIP with dual technical contributions: 1) A dataset reconstruction framework that performs semantic-constrained action unit segmentation and re-annotation on open-source robotic videos, constructing purified training sets containing singular atomic actions (e.g., "grasp"); 2) A temporal-decoupling fine-tuning strategy based on Contrastive Language-Image Pretraining (CLIP) architecture, which disentangles temporal action features across video frames from object-centric characteristics to achieve hierarchical representation learning of robotic atomic actions.Experimental results in simulated environments demonstrate that the RoboAct-CLIP pretrained model achieves a 12% higher success rate than baseline VLMs, along with superior generalization in multi-object manipulation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
