Convolutional video models have an order of magnitude larger computational complexity than their counterpart image-level models. Constrained by computational resources, there is no model or training method that can train long video sequences end-to-end. Currently, the main-stream method is to split a raw video into clips, leading to incomplete fragmentary temporal information flow. Inspired by natural language processing techniques dealing with long sentences, we propose to treat videos as serial fragments satisfying Markov property, and train it as a whole by progressively propagating information through the temporal dimension in multiple steps. This progressive training (PGT) method is able to train long videos end-to-end with limited resources and ensures the effective transmission of information. As a general and robust training method, we empirically demonstrate that it yields significant performance improvements on different models and datasets. As an illustrative example, the proposed method improves SlowOnly network by 3.7 mAP on Charades and 1.9 top-1 accuracy on Kinetics with negligible parameter and computation overhead. Code is available at https://github.com/BoPang1996/PGT.
Progressively Deformable 2D Gaussian Splatting for Video Representation at Arbitrary Resolutions
Shanghai Jiao Tong University, Shanghai, China(上海交通大学);University of Missouri--Kansas City, Kansas City, MO, USA(密苏里大学-堪萨斯城分校);University of Canterbury, Christchurch, New Zealand(坎特伯雷大学)
PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models
Tsinghua University(清华大学);OpenGVLab(开放通用实验室);Shanghai AI Laboratory(上海人工智能实验室);SenseTime Research(商汤科技研究院);Nanjing University(南京大学);The Chinese University of Hong Kong(香港中文大学)
PSVT: End-to-End Multi-person 3D Pose and Shape Estimation with Progressive Video Transformers
University of Science and Technology Beijing(北京科技大学);University of Sydney(悉尼大学);Beijing Engineering Research Center of Industrial Spectrum Imaging(北京工业光谱成像工程研究中心);Baidu(百度)
PMQ-VE: Progressive Multi-Frame Quantization for Video Enhancement
USTC(中国科学技术大学);Max Planck Institute(马克斯·普朗克研究所);CUHK(香港中文大学);SJTU(上海交通大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Chang’an University(长安大学)
Perceive, Reflect and Understand Long Video: Progressive Multi-Granular Clue Exploration with Interactive Agents
School of Electronic Engineering, Xidian University(西安电子科技大学电子工程学院);Department of Computer Science and Engineering, Hong Kong University of Science and Technology(香港理工大学计算机科学与工程系);College of Computer and Information, Hohai University(河海大学计算机与信息学院)
Light-A-Video: Training-free Video Relighting via Progressive Light Fusion
Shanghai Jiao Tong University(上海交通大学);University of Science and Technology of China(中国科学技术大学);The Chinese University of Hong Kong(香港中文大学);Hong Kong University of Science and Technology(香港科技大学);Stanford University(斯坦福大学);Shanghai AI Laboratory(上海人工智能实验室)
PRVQL: Progressive Knowledge-guided Refinement for Robust Egocentric Visual Query Localization
University of North Texas(北卡罗来纳州立大学);University of Texas at Dallas(德克萨斯大学达拉斯分校);U.S. Naval Research Laboratory(美国海军研究实验室);Brookhaven National Laboratory(布鲁克海文国家实验室)
Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding
Nanjing University of Science and Technology(南京理工大学);Ant Group(蚂蚁集团);National University of Singapore(新加坡国立大学);The Chinese University of Hong Kong(香港中文大学)
Chain-of-Glimpse: Search-Guided Progressive Object-Grounded Reasoning for Video Understanding
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China.(网络与交换技术国家重点实验室,北京邮电大学,北京,中国);Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, China.(大数据研究院,复旦大学计算机科学与人工智能学院,中国);ARC Lab, Tencent PCG, Shenzhen, China.(腾讯PCG深圳实验室,深圳,中国);School of Artificial Intelligence, Beijing University of Technology, Beijing, China.(北京理工大学人工智能学院,北京,中国)
4DGCPro: Efficient Hierarchical 4D Gaussian Compression for Progressive Volumetric Video Streaming
Cooperative Medianet Innovation Center, Shanghai Jiaotong University(合作中位网创新中心,上海交通大学);Department of Electronics, Shanghai Jiaotong University(电子工程系,上海交通大学);Shanghai AI Lab(上海人工智能实验室);Cloud platform department, E-surfing Vision Technology Co., Ltd.(云平台部门,E-surfing Vision Technology有限公司);School of Information Science and Technology, ShanghaiTech University(信息科学与技术学院,上海科技大学)
Error-Propagation-Free Learned Video Compression With Dual-Domain Progressive Temporal Alignment
Department of Electronic Engineering, Shanghai Jiao Tong University(上海交通大学电子工程系);Department of Computer Science and Engineering, Shanghai Jiao Tong University(上海交通大学计算机科学与工程系);College of Information Science and Electronic Engineering, Zhejiang University(浙江大学信息科学与电子工程学院);Naval Aviation University(海军航空大学)
VideoMem: Enhancing Ultra-Long Video Understanding via Adaptive Memory Management
School of Electronic and Computer Engineering, Peking University(电子与计算机工程学院,北京大学);Department of Computer Science and Technology, Tsinghua University(计算机科学与技术系,清华大学)
Conan: Progressive Learning to Reason Like a Detective over Multi-Scale Visual Evidence
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机学院,北京大学);WeChat AI, Tencent Inc., China(微信AI,腾讯公司,中国)
ProPhy: Progressive Physical Alignment for Dynamic World Simulation
Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Peng Cheng Laboratory(鹏城实验室);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);ETH Zürich(苏黎世联邦理工学院);Lenovo Research(联想研究院)
Efficient Training for Human Video Generation with Entropy-Guided Prioritized Progressive Learning
Stanford University(斯坦福大学);North China Electric Power University(华北电力大学);University of Adelaide(阿德莱德大学);University of Technology Sydney(悉尼技术大学);University of Science and Technology of China(中国科学技术大学)
TunerDiT: Training-free Progressive Steering of Diffusion Transformer for Multi-Event Video Generation
Ludwig Maximilian University of Munich(慕尼黑路德维希-马克西米利安大学);Technical University of Munich(慕尼黑技术大学);MCML;University of Hamburg(汉堡大学);Huawei European Research Institute(华为欧洲研究院)
Progressive Image Restoration via Text-Conditioned Video Generation
Department of Computer Science, University of Illinois Springfield(伊利诺伊大学斯普林菲尔德分校计算机科学系);School of Electrical and Computer Engineering, Purdue University(普渡大学电气与计算机工程学院)
Artemis: Geometry-Grounded Multi-Agent Driving World Models with Shared 3D State and Progressive Memory Update
University of Oxford(牛津大学);University of Cambridge(剑桥大学);University of Twente(特文特大学);National University of Singapore(新加坡国立大学);University of Bath(巴斯大学)