DexImit：从单目人类视频中学习双臂灵巧操作
DexImit: Learning Bimanual Dexterous Manipulation from Monocular Human Videos
AI总结：
DexImit通过四阶段生成流程，将单目人类视频转化为物理合理的机器人数据，以解决双臂灵巧操作的泛化问题。
摘要：
数据稀缺从根本上限制了双臂灵巧操作的泛化能力，因为真实世界中收集灵巧手数据成本高且劳动密集。人类操作视频作为一种直接承载操作知识的载体，为机器人学习的扩展提供了巨大潜力。然而，人类手与机械灵巧手之间显著的具身差距使得直接从人类视频中预训练极为困难。为弥合这一差距并释放大规模人类操作视频数据的潜力，我们提出DexImit，一个自动化框架，将单目人类操作视频转换为物理上合理的机器人数据，而无需任何额外信息。DexImit采用四阶段生成流程：（1）从任意视角重建手-物体交互，具有近米级尺度；（2）执行子任务分解和双臂调度；（3）合成与演示交互一致的机器人轨迹；（4）全面的数据增强以实现零样本现实部署。基于这些设计，DexImit可以根据人类视频生成大规模机器人数据，无论是从互联网还是视频生成模型中获取。DexImit能够处理多种操作任务，包括工具使用（例如切苹果）、长周期任务（例如制作饮料）和精细操作（例如叠杯子）。
英文摘要：
Data scarcity fundamentally limits the generalization of bimanual dexterous manipulation, as real-world data collection for dexterous hands is expensive and labor-intensive. Human manipulation videos, as a direct carrier of manipulation knowledge, offer significant potential for scaling up robot learning. However, the substantial embodiment gap between human hands and robotic dexterous hands makes direct pretraining from human videos extremely challenging. To bridge this gap and unleash the potential of large-scale human manipulation video data, we propose DexImit, an automated framework that converts monocular human manipulation videos into physically plausible robot data, without any additional information. DexImit employs a four-stage generation pipeline: (1) reconstructing hand-object interactions from arbitrary viewpoints with near-metric scale; (2) performing subtask decomposition and bimanual scheduling; (3) synthesizing robot trajectories consistent with the demonstrated interactions; (4) comprehensive data augmentation for zero-shot real-world deployment. Building on these designs, DexImit can generate large-scale robot data based on human videos, either from the Internet or video generation models. DexImit is capable of handling diverse manipulation tasks, including tool use (e.g., cutting an apple), long-horizon tasks (e.g., making a beverage), and fine-grained manipulations (e.g., stacking cups).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
