Haoran Lu, Shang Wu, Songling Liu, Jianshu Zhang, Maojiang Su, Guo Ye, Chenwei Xu, Lie Lu, Pranav Maneriker, Fan Du, Zhaoran Wang, Han Liu
备注:
v2:Expanded the experiment section with more baselines and add more experiments in supplementary--corrected some typographical errors, and corrected author-affiliation information that was inaccurate in the previous version
Recent video diffusion models have achieved impressive capabilities as large-scale generative world models. However, these models often struggle with fine-grained physical consistency, exhibiting physically implausible dynamics over time. In this work, we present \textbf{Phys4D}, a pipeline for learning physics-consistent 4D world representations from video diffusion models. Phys4D adopts \textbf{a three-stage training paradigm} that progressively lifts appearance-driven video diffusion models into physics-consistent 4D world representations. We first bootstrap robust geometry and motion representations through large-scale pseudo-supervised pretraining, establishing a foundation for 4D scene modeling. We then perform physics-grounded supervised fine-tuning using simulation-generated data, enforcing temporally consistent 4D dynamics. Finally, we apply simulation-grounded reinforcement learning to correct residual physical violations that are difficult to capture through explicit supervision. To evaluate fine-grained physical consistency beyond appearance-based metrics, we introduce a set of \textbf{4D world consistency evaluation} that probe geometric coherence, motion stability, and long-horizon physical plausibility. Experimental results demonstrate that Phys4D substantially improves fine-grained spatiotemporal and physical consistency compared to appearance-driven baselines, while maintaining strong generative performance. Our project page is available at https://sensational-brioche-7657e7.netlify.app/
FiVE: A Fine-grained Video Editing Benchmark for Evaluating Emerging Diffusion and Rectified Flow Models
Harvard AI and Robotics Lab, Harvard University(哈佛人工智能与机器人实验室,哈佛大学);Broad Institute(博德研究所);Hong Kong Polytechnic University(香港理工大学);School of Engineering and Applied Sciences, Harvard University(哈佛大学工程与应用科学学院);City University of Hong Kong(香港城市大学);Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University(自然与人工智能研究学院,哈佛大学)
OpenNLPLab(开放自然语言处理实验室);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Hefei University of Technology(合肥工业大学);Northwestern Polytechnical University(西北工业大学);The University of Hong Kong(香港大学)
Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models
University of California, Davis(加州大学戴维斯分校);Virginia Tech(弗吉尼亚理工大学);The Chinese University of Hong Kong(香港中文大学);NVIDIA;Adobe Research(Adobe研究);Fudan University(复旦大学);Meta AI
ViSeRet: A simple yet effective approach to moment retrieval via fine-grained video segmentation
Twelve Labs;KAIST(韩国科学技术院)
视频文本检索有许多现实世界的应用,如媒体分析,监控和机器人。本文介绍了ICCV Value Challenge 2021视频检索赛道的第一名解决方案。我们提出了一个简单而有效的方法来联合处理两个视频文本检索任务(视频检索和视频语料库矩检索),利用模型只训练视频检索任务。此外,我们还创建了一个集成模型,在Value Challenge中提出的所有四个数据集(TVr,How2r,YouCook2r和VATEXr)上实现了最先进的性能。
Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering
School of Information Science and Technology, Hangzhou Normal University(杭州师范大学信息科学与技术学院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);School of Computer Science and Technology, University of Chinese Academy of Sciences(中国科学院大学计算机科学与技术学院);School of Computer Science and Technology, Hangzhou Dianzi University(杭州电子科技大学计算机科学与技术学院)
PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs
Zhejiang University(浙江大学);ZJU-Hangzhou Global Scientific and Technological Innovation Center(浙江大学杭州国际科技创新中心);Mohamed Bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Exploiting Feature Diversity for Make-up Temporal Video Grounding
Tencent Youtu Lab(腾讯优图实验室)
本技术报告介绍了MTVG的第三个获奖解决方案,这是在ACM MM 2022的第四届情景中人(PIC)挑战赛中引入的一项新任务。MTVG的目的是在一个未修剪的视频的基础上的文本描述的时间边界的步骤。这项任务的最大挑战是化妆步骤的细粒度视频-文本语义。然而,目前的方法主要是使用基于动作的预训练模型提取视频特征。由于动作比化妆步骤更粗粒度,因此基于动作的特征不足以提供细粒度的线索。为了解决这个问题,我们建议通过利用功能diffusion实现细粒度表示。具体来说,我们提出了一系列的方法,从特征提取,网络优化,模型集成。因此,我们在MTVG比赛中获得了第三名。
EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching
University of Science and Technology of China(中国科学技术大学);Southeast University(东南大学);Alibaba Group(阿里巴巴集团);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);CAS Center for Excellence in Brain Science and Intelligence Technology(中国科学院脑科学与智能技术卓越创新中心)
Sports Video: Fine-Grained Action Detection and Classification of Table Tennis Strokes from Videos for MediaEval 2021
Max Planck Institute for Evolutionary Anthropology(马克斯·普朗克进化人类学研究所);La Rochelle University(拉罗谢尔大学);Univ. Bordeaux(波尔多大学);CNRS(法国国家科学研究中心);Bordeaux INP(波尔多国立高等理工学院);LaBRI(波尔多计算机科学实验室);University of Bordeaux(波尔多大学)