arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.24285cs.RO
收藏

ACTS:从人类物理交互中学习视觉-触觉世界模型

ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction

Massachusetts Institute of Technology(麻省理工学院);University of Washington(华盛顿大学);Harvard University(哈佛大学);University of California, Berkeley(加州大学伯克利分校);Stanford University(斯坦福大学)

Yuxiang Ma, Weihan Xu, Pengfei Ye, Anna Mei, Yuxin Ray Song, Tingle Li, Haonan Chen, Edward H. Adelson

AI总结:

本文提出仅基于人类交互数据训练的视觉-触觉世界模型ACTS,将人类接触动力学迁移至机器人,发现接触动力学迁移比视觉外观更可靠。

摘要:

针对接触丰富的操纵任务的动作条件世界模型,需要能捕捉不同力作用下接触形成、滑动和释放过程的数据。这类数据通过机器人遥操作采集速度缓慢,且常缺乏力反馈,还受限于夹具几何结构。本文探究能否从人类物理交互中学习这些动力学并迁移至机器人。我们提出ACTS,这是一种仅基于人类交互数据训练的视觉-触觉世界模型。操作者操控手持GelSight传感器,我们记录第三人称视角和手腕视角视频、触觉图像、6自由度传感器运动以及触觉导出的法向力,得到React数据集,这是一个时长4.9小时的双操作臂数据集,涵盖三个物体。我们将触觉传感器的动作表示为运动与力变化的结合,为人类采集和机器人执行提供共享动作空间。ACTS联合预测未来场景、手腕和触觉观测结果。在保留的人类数据上,ACTS在场景视角预测的峰值信噪比(PSNR)上,比在React上从头训练的两个视觉-触觉世界模型中更强的那个提升了0.7至2.0分贝。在未微调的机器人遥操作部署中,我们观测到,预测的触觉印记仍会随记录的接触出现和消失。视觉迁移的鲁棒性较弱:在PushT数据集上,相机预测优于帧复制;在Rope数据集上表现不一;在Motherboard数据集上则低于帧复制;而手腕视角可能模糊或生成虚假结构。这些结果表明,接触动力学从人类到机器人的迁移比视觉外观更可靠。

英文摘要:

Abstract Action-conditioned world models for contact-rich manipulation require data that captures how contact forms, slides, and releases under varying force. Such data are slow to collect through robot teleoperation, which often lacks force feedback and is constrained by gripper geometry. We ask whether these dynamics can instead be learned from human physical interaction and transferred to a robot. We present ACTS, a visuo-tactile world model trained only on human interaction data. Operators manipulate handheld GelSight sensors while we record third-person and wrist video, tactile images, 6-DoF sensor motion, and tactile-derived normal force, yielding React, a 4.9-hour bimanual dataset across three objects. We represent actions at the tactile sensor as motion augmented by force change, providing a shared action space for human collection and robot execution. ACTS jointly predicts future scene, wrist, and tactile observations. On held-out human data, ACTS improves scene-view prediction by 0.7-2.0 dB PSNR over the stronger of two visuo-tactile world models trained from scratch on React. Without fine-tuning, in the robot teleoperation rollouts we inspected, predicted tactile imprints still appear and disappear with recorded contact. Visual transfer is less robust: camera predictions exceed frame copying on PushT, are mixed on Rope, and fall below it on Motherboard, while wrist views can blur or hallucinate structure. These results suggest that contact dynamics transfer more reliably than visual appearance from human to robot.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

相关深度报道

↑