TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning
Nanjing University(南京大学);Shanghai AI Laboratory(上海人工智能实验室);SIAT, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Zhejiang University(浙江大学);University of Science and Technology of China(中国科学技术大学);Shanghai Jiao Tong University(上海交通大学);Fudan University(复旦大学)
Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in short video understanding. However, understanding long-form videos still remains challenging for MLLMs. This paper proposes TimeSuite, a collection of new designs to adapt the existing short-form video MLLMs for long video understanding, including a simple yet efficient framework to process long video sequence, a high-quality video dataset for grounded tuning of MLLMs, and a carefully-designed instruction tuning task to explicitly incorporate the grounding supervision in the traditional QA format. Specifically, based on VideoChat, we propose our long-video MLLM, coined as VideoChat-T, by implementing a token shuffling to compress long video tokens and introducing Temporal Adaptive Position Encoding (TAPE) to enhance the temporal awareness of visual representation. Meanwhile, we introduce the TimePro, a comprehensive grounding-centric instruction tuning dataset composed of 9 tasks and 349k high-quality grounded annotations. Notably, we design a new instruction tuning task type, called Temporal Grounded Caption, to peform detailed video descriptions with the corresponding time stamps prediction. This explicit temporal location prediction will guide MLLM to correctly attend on the visual content when generating description, and thus reduce the hallucination risk caused by the LLMs. Experimental results demonstrate that our TimeSuite provides a successful solution to enhance the long video understanding capability of short-form MLLM, achieving improvement of 5.6% and 6.8% on the benchmarks of Egoschema and VideoMME, respectively. In addition, VideoChat-T exhibits robust zero-shot temporal grounding capabilities, significantly outperforming the existing state-of-the-art MLLMs. After fine-tuning, it performs on par with the traditional supervised expert models.
MELLA: Bridging Linguistic Capability and Cultural Groundedness for Low-Resource Language MLLMs
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);East China Normal University(东华大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Institute of High Performance Computing, A*STAR(高性能计算研究所,A*STAR)
Concept-Grounded Reasoning with Prompt-Driven Localization for Interpretable Structured Report Generation
The Hong Kong University of Science and Technology(香港科技大学);The Chinese University of Hong Kong(香港中文大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Saarland University(萨尔兰大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of Cambridge(剑桥大学)
Watch, Remember, Reason: Human-View Video Understanding with MLLMs
School of Intelligence Science and Technology, Peking University(北京理工大学智能科学与技术学院);Wuhan University(武汉大学);Shanghai Jiao Tong University(上海交通大学);Nanyang Technological University(南洋理工大学);CASIA(中国科学院自动化研究所);University of Tokyo(东京大学);University of Liverpool(利物浦大学);Zhejiang University(浙江大学);National University of Singapore(新加坡国立大学);UC Merced(加州大学默塞德分校)
Groc-PO: Grounded Context Preference Optimization for Truthful Multimodal LLMs
University of Science and Technology of China(中国科学技术大学);Xiaomi Corporation(小米公司);State Key Laboratory of Communication Content Cognition, People’s Daily Online(人民日报社传播内容认知国家重点实验室)
I Spy With My Model's Eye: Visual Search as a Behavioural Test for MLLMs
Leverhulme Centre for the Future of Intelligence, University of Cambridge(未来智能研究中心、剑桥大学);Department of Engineering, University of Cambridge(工程系、剑桥大学);Department of Psychology, University of Cambridge(心理学系、剑桥大学);Department of Computer Science, University of Cambridge(计算机科学系、剑桥大学)
TableVision: A Large-Scale Benchmark for Spatially Grounded Reasoning over Complex Hierarchical Tables
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学)
GeoEyes: On-Demand Visual Focusing for Evidence-Grounded Understanding of Ultra-High-Resolution Remote Sensing Imagery
National University of Defense Technology, China(国防科技大学);Beijing University of Posts and Telecommunications, China(北京邮电大学);University of the Chinese Academy of Sciences, China(中国科学院大学);Shanghai Jiao Tong University, China(上海交通大学);Wuhan University, China(武汉大学);Chinese Academy of Science, China(中国科学院);Tsinghua University, China(清华大学);Renmin University of China, China(中国人民大学)
ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs
Université de Montréal(蒙特利尔大学);Mila – Quebec AI Institute(米拉-魁北克人工智能研究所);McGill University(麦吉尔大学);McMaster University(麦克马斯特大学);Meta Platforms(Meta平台公司)
CGC: Compositional Grounded Contrast for Fine-Grained Multi-Image Understanding
School of Computer Science and Technology, Hangzhou Dianzi University(杭州电子科技大学计算机科学与技术学院);School of Computer Science(计算机科学学院);Technology, Hangzhou Dianzi University(技术,杭州电子科技大学)
Beyond RGB: Benchmarking and Enhancing MLLMs for Hyperspectral Image Understanding via Training-Free Reasoning Framework
Sun Yat-sen University(中山大学);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);China Agricultural University(中国农业大学);Southwest Jiaotong University(西南交通大学);Southwest University(西南大学);National Supercomputing Center in Shenzhen(国家超级计算深圳中心)
DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning
Department of Automation, Tsinghua University, Beijing, China(清华大学自动化系);Zhongguancun Academy, Beijing, China(中关村学院);School of Software, Xinjiang University, Urumqi, China(新疆大学软件学院);College of Materials Science and Engineering, Fuzhou University, Fuzhou, China(福州大学材料科学与工程学院);Institute of Automation, Chinese Academy of Sciences, Beijing, China(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China(中国科学院大学人工智能学院);Beijing Qianjue Technology Co., Ltd., Beijing, China(北京千 jue 技术有限公司)