arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.10448cs.RO
收藏

DiT4DiT:联合建模视频动态与动作以实现通用机器人控制

DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control

Mondo Robotics(Mondo机器人公司);HKUST(GZ)(香港科技大学(广州));HKUST(香港科技大学)

Teli Ma, Jia Zheng, Zifan Wang, Chunli Jiang, Andy Cui, Junwei Liang, Shuo Yang

AI总结:

本文提出DiT4DiT模型,通过结合视频扩散变换器与动作扩散变换器,实现视频动态与动作的联合建模,提升机器人控制的泛化能力与样本效率。

摘要:

本文提出DiT4DiT模型,通过结合视频扩散变换器与动作扩散变换器,实现视频动态与动作的联合建模,提升机器人控制的泛化能力与样本效率。

英文摘要:

Vision-Language-Action (VLA) models have emerged as a promising paradigm for robot learning, but their representations are still largely inherited from static image-text pretraining, leaving physical dynamics to be learned from comparatively limited action data. Generative video models, by contrast, encode rich spatiotemporal structure and implicit physics, making them a compelling foundation for robotic manipulation. But their potentials are not fully explored in the literature. To bridge the gap, we introduce DiT4DiT, an end-to-end Video-Action Model that couples a video Diffusion Transformer with an action Diffusion Transformer in a unified cascaded framework. Instead of relying on reconstructed future frames, DiT4DiT extracts intermediate denoising features from the video generation process and uses them as temporally grounded conditions for action prediction. We further propose a dual flow-matching objective with decoupled timesteps and noise scales for video prediction, hidden-state extraction, and action inference, enabling coherent joint training of both modules. Across simulation and real-world benchmarks, DiT4DiT achieves state-of-the-art results, reaching average success rates of 98.6% on LIBERO and 50.8% on RoboCasa GR1 while using substantially less training data. On the Unitree G1 robot, it also delivers superior real-world performance and strong zero-shot generalization. Importantly, DiT4DiT improves sample efficiency by over 10x and speeds up convergence by up to 7x, demonstrating that video generation can serve as an effective scaling proxy for robot policy learning. We release code and models at https://dit4dit.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑