University of Waterloo(滑铁卢大学);Georgia Institute of Technology(佐治亚理工学院);Stanford University(斯坦福大学);Google Deepmind(谷歌DeepMind);New York University(纽约大学)
Achint Soni, Sreyas Venkataraman, Abhranil Chandra, Sebastian Fischmeister, Percy Liang, Bo Dai, Sherry Yang
Video generation has been used to generate visual plans for controlling robotic systems. Given an image observation and a language instruction, previous work has generated video plans which are then converted to robot controls to be executed. However, a major bottleneck in leveraging video generation for control lies in the quality of the generated videos, which often suffer from hallucinatory content and unrealistic physics, resulting in low task success when control actions are extracted from the generated videos. While scaling up dataset and model size provides a partial solution, integrating external feedback is both natural and essential for grounding video generation in the real world. With this observation, we propose VideoAgent for self-improving generated video plans based on external feedback. Instead of directly executing the generated video plan, VideoAgent first refines the generated video plans using a novel procedure which we call self-conditioning consistency, allowing inference-time compute to be turned into better generated video plans. As the refined video plan is being executed, VideoAgent can collect additional data from the environment to further improve video plan generation. Experiments in simulated robotic manipulation from MetaWorld and iTHOR show that VideoAgent drastically reduces hallucination, thereby boosting success rate of downstream manipulation tasks. We further illustrate that VideoAgent can effectively refine real-robot videos, providing an early indicator that robots can be an effective tool in grounding video generation in the physical world. Video demos and code can be found at https://video-as-agent.github.io.
CanvasMAR: Improving Masked Autoregressive Video Prediction With Canvas
Institute for Artificial Intelligence, Peking University, Beijing, China(人工智能研究院,北京大学,北京,中国);School of Intelligence Science and Technology, Peking University, Beijing, China(智能科学与技术学院,北京大学,北京,中国);State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China(通用人工智能国家重点实验室,北京大学,北京,中国)
CausalChapter: Improving Long-Video Chaptering with Interventional Dependency Modeling
School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院);Beijing Key Laboratory of Artificial Intelligence for Education(北京市教育人工智能重点实验室);Engineering Research Center of Intelligent Technology and Educational Application, Ministry of Education(教育部智能技术与教育应用工程研究中心)
TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning
Nanjing University(南京大学);Shanghai AI Laboratory(上海人工智能实验室);SIAT, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Zhejiang University(浙江大学);University of Science and Technology of China(中国科学技术大学);Shanghai Jiao Tong University(上海交通大学);Fudan University(复旦大学)
CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoising
Sun Yat-sen University(中山大学);The Key Laboratory of Machine Intelligence and Advanced Computing (Sun Yat-sen University), Ministry of Education(机器智能与先进计算教育部重点实验室(中山大学));JD AI Research(京东人工智能研究院)