TASTE-Rob：推进面向任务的手-物交互视频生成，以实现可泛化的机器人操作
TASTE-Rob: Advancing Video Generation of Task-Oriented Hand-Object Interaction for Generalizable Robotic Manipulation
AI总结：
本文构建含100,856个对齐语言指令的第一视角手-物交互视频数据集 TASTE-Rob，并结合VDM微调和三阶段手部姿势精修流程，提升面向机器人模仿学习的可泛化操作视频生成质量。
摘要：
我们解决了现有面向任务的手-物交互视频生成数据集和模型中的关键局限，这类视频生成是为机器人模仿学习生成视频演示的一种重要方法。当前数据集（例如 Ego4D）往往存在视角不一致和交互未对齐的问题，导致视频质量下降，并限制了其在精确模仿学习任务中的适用性。为此，我们推出 TASTE-Rob——一个开创性的大规模数据集，包含 100,856 个第一视角手-物交互视频。每个视频都与语言指令精心对齐，并从一致的相机视角录制，以确保交互清晰。通过在 TASTE-Rob 上微调视频扩散模型（VDM），我们实现了逼真的物体交互，但也观察到手部抓握姿势偶尔存在不一致。为增强真实感，我们提出一个三阶段姿势精修流程，用于提高生成视频中的手部姿势准确性。我们精心构建的数据集与专门的姿势精修框架相结合，在生成高质量、面向任务的手-物交互视频方面带来显著性能提升，从而实现更优的可泛化机器人操作。TASTE-Rob 数据集公开发布以推动该领域进一步发展，TASTE-Rob 数据集和源代码将在我们的网站 https://taste-rob.github.io 上公开提供。
英文摘要：
We address key limitations in existing datasets and models for task-oriented hand-object interaction video generation, a critical approach of generating video demonstrations for robotic imitation learning. Current datasets, such as Ego4D, often suffer from inconsistent view perspectives and misaligned interactions, leading to reduced video quality and limiting their applicability for precise imitation learning tasks. Towards this end, we introduce TASTE-Rob -- a pioneering large-scale dataset of 100,856 ego-centric hand-object interaction videos. Each video is meticulously aligned with language instructions and recorded from a consistent camera viewpoint to ensure interaction clarity. By fine-tuning a Video Diffusion Model (VDM) on TASTE-Rob, we achieve realistic object interactions, though we observed occasional inconsistencies in hand grasping postures. To enhance realism, we introduce a three-stage pose-refinement pipeline that improves hand posture accuracy in generated videos. Our curated dataset, coupled with the specialized pose-refinement framework, provides notable performance gains in generating high-quality, task-oriented hand-object interaction videos, resulting in achieving superior generalizable robotic manipulation. The TASTE-Rob dataset is publicly available to foster further advancements in the field, TASTE-Rob dataset and source code will be made publicly available on our website https://taste-rob.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
