Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task. In reality, a truly useful VidL system is expected to be easily generalizable to diverse tasks, domains, and datasets. To facilitate the evaluation of such systems, we introduce Video-And-Language Understanding Evaluation (VALUE) benchmark, an assemblage of 11 VidL datasets over 3 popular tasks: (i) text-to-video retrieval; (ii) video question answering; and (iii) video captioning. VALUE benchmark aims to cover a broad range of video genres, video lengths, data volumes, and task difficulty levels. Rather than focusing on single-channel videos with visual information only, VALUE promotes models that leverage information from both video frames and their associated subtitles, as well as models that share knowledge across multiple tasks. We evaluate various baseline methods with and without large-scale VidL pre-training, and systematically investigate the impact of video input channels, fusion methods, and different video representations. We also study the transferability between tasks, and conduct multi-task learning under different settings. The significant gap between our best model and human performance calls for future study for advanced VidL models. VALUE is available at https://value-benchmark.github.io/.
VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitions
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院);National Key Laboratory of General Artificial Intelligence(通用人工智能全国重点实验室)
VideoExplorer: Think With Videos For Agentic Long-Video Understanding
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Beijing University of Posts and Telecommunications(北京邮电大学);Hong Kong Polytechnic University(香港理工大学);Peking University(北京大学);University of Trento(特伦托大学)
Video-CoT: A Comprehensive Dataset for Spatiotemporal Understanding of Videos Based on Chain-of-Thought
Institute of Automation, CAS(中国科学院自动化研究所);School of Artifcial Intelligence, UCAS(中国科学技术大学人工智能学院);Beijing Academy of Artificial Intelligence (BAAI)(北京人工智能研究院);Shenzhen International GraduateSchool,Tsinghua University(深圳国际研究生院,清华大学);State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,北京大学计算机学院)
PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs
Zhejiang University(浙江大学);ZJU-Hangzhou Global Scientific and Technological Innovation Center(浙江大学杭州国际科技创新中心);Mohamed Bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding
The Chinese University of Hong Kong(香港中文大学);Ant Group(蚂蚁集团);Tsinghua University(清华大学);Tongji University(同济大学);The Hong Kong University of Science and Technology(香港科技大学)
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);OpenGVLab, Shanghai AI Laboratory(上海人工智能实验室 OpenGVLab);The University of Hong Kong(香港大学);Nanjing University(南京大学)
VideoMamba: State Space Model for Efficient Video Understanding
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);OpenGVLab, Shanghai AI Laboratory(上海人工智能实验室 OpenGVLab)
MLVU: Benchmarking Multi-task Long Video Understanding
Beijing Academy of Artificial Intelligence(北京人工智能研究院);Beijing University of Posts and Telecommunications(北京邮电大学);Shanghai Jiao Tong University(上海交通大学);Zhejiang University(浙江大学);Peking University(北京大学)
1st Place Winner of the 2024 Pixel-level Video Understanding in the Wild (CVPR'24 PVUW) Challenge in Video Panoptic Segmentation and Best Long Video Consistency of Video Semantic Segmentation
Device Solutions Research America, Samsung Semiconductor, Inc.(三星半导体公司美国设备解决方案研究中心)
Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos
Bytedance Inc.(字节跳动公司);AAII, UTS(悉尼科技大学人工智能研究院);MBZUAI(穆罕默德·本·扎耶德人工智能大学);University of Science and Technology of China(中国科学技术大学);Data61, CSIRO(澳大利亚联邦科学与工业研究组织数据61部门)
v-HUB: A Benchmark for Video Humor Understanding from Vision and Sound
Shanghai Jiao Tong University(上海交通大学);Wuhan University(武汉大学);Beijing Institute for General Artificial Intelligence(北京一般人工智能研究院);Independent Researcher(独立研究者)
VideoKR: Towards Knowledge- and Reasoning-Intensive Video Understanding
University of California, Berkeley(加州大学伯克利分校);Stanford University(斯坦福大学);University of Toronto(多伦多大学);University of Washington(华盛顿大学);University of Michigan(密歇根大学)