Multi-modal transformers are rapidly gaining attention in video captioning tasks. Existing multi-modal video captioning methods typically extract a fixed number of frames, which raises critical challenges. When a limited number of frames are extracted, important frames with essential information for caption generation may be missed. Conversely, extracting an excessive number of frames includes consecutive frames, potentially causing redundancy in visual tokens extracted from consecutive video frames. To extract an appropriate number of frames for each video, this paper proposes the first model-agnostic module selection framework in video captioning that has two main functions: (1) selecting a caption generation module with an appropriate size based on visual tokens extracted from video frames, and (2) constructing subsets of visual tokens for the selected caption generation module. Furthermore, we propose a new adaptive attention masking scheme that enhances attention on important visual tokens. Our experiments on three different benchmark datasets demonstrate that the proposed framework significantly improves the performance of three recent video captioning models.
VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generation
New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences (CASIA)（模式识别新技术实验室，自动化研究所，中国科学院（CASIA））；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Kuaishou Technology（快手科技）；Nanjing University（南京大学）
We propose UniTAB that Unifies Text And Box outputs for grounded vision-language (VL) modeling. Grounded VL tasks such as grounded captioning require the model to generate a text description and align predicted words with object regions. To achieve this, models must generate desired text and box outputs together, and meanwhile indicate the alignments between words and boxes. In contrast to existing solutions that use multiple separate modules for different outputs, UniTAB represents both text and box outputs with a shared token sequence, and introduces a special <obj> token to naturally indica…
In this work, we pursue a unified paradigm for multimodal pretraining to break the scaffolds of complex task/modality-specific customization. We propose OFA, a Task-Agnostic and Modality-Agnostic framework that supports Task Comprehensiveness. OFA unifies a diverse set of cross-modal and unimodal tasks, including image generation, visual grounding, image captioning, image classification, language modeling, etc., in a simple sequence-to-sequence learning framework. OFA follows the instruction-based learning in both pretraining and finetuning stages, requiring no extra task-specific layers for d…
Visual knowledge bases such as Visual Genome power numerous applications in computer vision, including visual question answering and captioning, but suffer from sparse, incomplete relationships. All scene graph models to date are limited to training on a small set of visual relationships that have thousands of training labels each. Hiring human annotators is expensive, and using textual knowledge base completion methods are incompatible with visual data. In this paper, we introduce a semi-supervised method that assigns probabilistic relationship labels to a large number of unlabeled images usi…
Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localization
MBZUAI；Provable Responsible AI and Data Analytics (PRADA) Lab（可证明负责任的人工智能与数据分析实验室）；King Abdullah University of Science and Technology（卡迪夫大学科学与技术大学）；School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉科学学院）；University of Copenhagen（哥本哈根大学）；Tsinghua University（清华大学）
At present, adversarial attacks are designed in a task-specific fashion. However, for downstream computer vision tasks such as image captioning, image segmentation etc., the current deep learning systems use an image classifier like VGG16, ResNet50, Inception-v3 etc. as a feature extractor. Keeping this in mind, we propose Mimic and Fool, a task agnostic adversarial attack. Given a feature extractor, the proposed attack finds an adversarial image which can mimic the image feature of the original image. This ensures that the two images give the same (or similar) output regardless of the task. W…
Universal Skeleton Understanding via Differentiable Rendering and MLLMs
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China（人工智能通用基础理论国家重点实验室，北京大学深圳研究生院，中国）；Tencent（腾讯）；Nanjing University of Aeronautics（南京航空航天大学）
VOST-SGG: VLM-Aided One-Stage Spatio-Temporal Scene Graph Generation
College of Computing and Data Science, Nanyang Technological University, Singapore（南洋理工大学计算与数据科学学院）；Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore（科学、技术与研究局高性能计算研究所）；Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore（科学、技术与研究局前沿人工智能研究中心）
Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Aalto University（阿尔托大学）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）
See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignment
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE（穆罕默德·本·扎耶德人工智能大学，阿布扎比，阿联酋）；King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia（法赫德国王石油矿产大学，达兰，沙特阿拉伯）；Macquarie University, Sydney, Australia（麦考瑞大学，悉尼，澳大利亚）；University of Surrey, Guildford, United Kingdom（萨里大学，吉尔福德，英国）；University of New South Wales, Sydney, Australia（新南威尔士大学，悉尼，澳大利亚）
The VALUE (Video-And-Language Understanding Evaluation) benchmark is newly introduced to evaluate and analyze multi-modal representation learning algorithms on three video-and-language tasks: Retrieval, QA, and Captioning. The main objective of the VALUE challenge is to train a task-agnostic model that is simultaneously applicable for various tasks with different characteristics. This technical report describes our winning strategies for the VALUE challenge: 1) single model optimization, 2) transfer learning with visual concepts, and 3) task-aware ensemble. The first and third strategies are d…
