While Omni-modal Large Language Models (OLLMs) have demonstrated impressive capabilities in jointly processing audio and visual streams, their ability to strictly adhere to complex, multi-faceted user instructions remains largely unexplored. Existing benchmarks primarily focus on holistic video understanding or text-only instruction following, failing to capture the intricate interplay between modalities and user constraints. To bridge this gap, we introduce OmniCap-IF, the first comprehensive benchmark specifically designed to evaluate instruction-following capabilities in omni-modal captioning. OmniCap-IF incorporates a systematic framework that assesses captions on two dimensions: format correctness and content correctness. Our benchmark encompasses 50 distinct constraint types across pure visual, pure audio, and audio-visual modalities, while integrating Temporal Grounding to assess spatio-temporal precision. Extensive evaluations of prominent models on 1,920 high-quality samples reveal significant performance disparities. Furthermore, our analysis uncovers a critical "format-content tradeoff", demonstrating that increasing formatting complexity directly degrades models' omni-modal reasoning abilities. Finally, to advance the field, we curate a 54K instruction-tuning dataset, OmniCap-IF-54K and present OmniCaptioner-IF, which achieves notable improvements in both complex instruction adherence and general omni-modal captioning performance.
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）
Shanghai Collaborative Innovation Center on Intelligent Visual Computing（上海智能视觉计算协同创新中心）；Microsoft Cloud + AI（微软云与人工智能部门）；Fudan University（复旦大学）；Microsoft Research Asia（微软亚洲研究院）
This paper presents OmniVL, a new foundation model to support both image-language and video-language tasks using one universal architecture. It adopts a unified transformer-based visual encoder for both image and video inputs, and thus can perform joint image-language and video-language pretraining. We demonstrate, for the first time, such a paradigm benefits both image and video tasks, as opposed to the conventional one-directional transfer (e.g., use image-language to help video-language). To this end, we propose a decoupled joint pretraining of image-language and video-language to effective…
OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Grounding
Wangxuan Institute of Computer Technology, Peking University（北京大学王轩计算机技术研究所）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）；Central Media Technology Institute, Huawei Technologies Ltd.（华为技术有限公司中央媒体技术研究所）；PKU-WUHAN Institute for Artificial Intelligence, Peking University（北京大学武汉人工智能研究所）
Microsoft Dynamics 365 AI Research（微软Dynamics 365人工智能研究院）
We present HERO, a novel framework for large-scale video+language omni-representation learning. HERO encodes multimodal inputs in a hierarchical structure, where local context of a video frame is captured by a Cross-modal Transformer via multimodal fusion, and global video context is captured by a Temporal Transformer. In addition to standard Masked Language Modeling (MLM) and Masked Frame Modeling (MFM) objectives, we design two new pre-training tasks: (i) Video-Subtitle Matching (VSM), where the model predicts both global and local temporal alignment; and (ii) Frame Order Modeling (FOM), whe…
LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos
Southern University of Science and Technology（南方科技大学）；University of Birmingham（伯明翰大学）；University of Electronic Science and Technology of China（电子科技大学）；The University of Hong Kong（香港大学）；University of Manchester（曼彻斯特大学）
OmniDiff: A Comprehensive Benchmark for Fine-grained Image Difference Captioning
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）
CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology
Westlake University（西湖大学）；Zhejiang University（浙江大学）；Harvard University（哈佛大学）；The Ohio State University（俄亥俄州立大学）；University of the Chinese Academy of Sciences（中国科学院大学）
LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning
Fudan University（复旦大学）；Tencent PCG（腾讯平台与内容事业群）；Centre for Frontier AI Research (CFAR), A*STAR（新加坡科技研究局前沿人工智能研究中心）；National University of Singapore（新加坡国立大学）
IBM Cloud（IBM云）；IBM Research（IBM研究院）；University of Amsterdam（阿姆斯特丹大学）
Transformer is a popularly used neural network architecture, especially for language understanding. We introduce an extended and unified architecture that can be used for tasks involving a variety of modalities like image, text, videos, etc. We propose a spatio-temporal cache mechanism that enables learning spatial dimension of the input in addition to the hidden states corresponding to the temporal input sequence. The proposed architecture further enables a single model to support tasks with multiple input modalities as well as asynchronous multi-task learning, thus we refer to it as OmniNet.…
Facebook AI Research（Facebook 人工智能研究院）；University of California, Berkeley（加利福尼亚大学伯克利分校）
Image descriptions can help visually impaired people to quickly understand the image content. While we made significant progress in automatically describing images and optical character recognition, current approaches are unable to include written text in their descriptions, although text is omnipresent in human environments and frequently critical to understand our surroundings. To study how to comprehend text in the context of an image we collect a novel dataset, TextCaps, with 145k captions for 28k images. Our dataset challenges a model to recognize text, relate it to its visual context, an…
Explore the Limits of Omni-modal Pretraining at Scale
The Chinese University of Hong Kong（香港中文大学）；University of Chinese Academy of Sciences（中国科学院大学）；Chinese Academy of Science（中国科学院）；Shanghai AI Laboratory（上海人工智能实验室）
VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Nanjing University of Science and Technology（南京理工大学）
