Multimodal Large Language Models (MLLMs) demonstrate remarkable image-language capabilities, but their widespread use faces challenges in cost-effective training and adaptation. Existing approaches often necessitate expensive language model retraining and limited adaptability. Additionally, the current focus on zero-shot performance improvements offers insufficient guidance for task-specific tuning. We propose CROME, an efficient vision-language instruction tuning framework. It features a novel gated cross-modal adapter that effectively combines visual and textual representations prior to input into a frozen LLM. This lightweight adapter, trained with minimal parameters, enables efficient cross-modal understanding. Notably, CROME demonstrates superior zero-shot performance on standard visual question answering and instruction-following benchmarks. Moreover, it yields fine-tuning with exceptional parameter efficiency, competing with task-specific specialist state-of-the-art methods. CROME demonstrates the potential of pre-LM alignment for building scalable, adaptable, and parameter-efficient multimodal models.
LLMs Can Evolve Continually on Modality for X-Modal Reasoning
Dalian University of Technology（大连理工大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology（香港科技大学）
Variational Adapter for Cross-modal Similarity Representation
School of Remote Sensing and Information Engineering（遥感与信息工程学院）；Wuhan University（武汉大学）；School of Data Science and Engineering（数据科学与工程学院）；East China Normal University（华东师范大学）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing（测绘遥感信息工程国家重点实验室）
Training-Free Dual Hyperbolic Adapters for Better Cross-Modal Reasoning
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；Department of Electrical and Electronic Engineering, Southern University of Science and Technology（南方科技大学电子与电气工程系）；Department of Applied Mathematics and Theoretical Physics, University of Cambridge（剑桥大学应用数学与理论物理系）；Yau Mathematical Sciences Center, Tsinghua University（清华大学应用数学中心）
Nanjing University（南京大学）；Shanghai AI Laboratory（上海人工智能实验室）
In this report, we present our champion solution to the WSDM2023 Toloka Visual Question Answering (VQA) Challenge. Different from the common VQA and visual grounding (VG) tasks, this challenge involves a more complex scenario, i.e. inferring and locating the object implicitly specified by the given interrogative question. For this task, we leverage ViT-Adapter, a pre-training-free adapter network, to adapt multi-modal pre-trained Uni-Perceiver for better cross-modal localization. Our method ranks first on the leaderboard, achieving 77.5 and 76.347 IoU on public and private test sets, respectiv…
Large-scale pretraining is fast becoming the norm in Vision-Language (VL) modeling. However, prevailing VL approaches are limited by the requirement for labeled data and the use of complex multi-step pretraining objectives. We present MAGMA - a simple method for augmenting generative language models with additional modalities using adapter-based finetuning. Building on Frozen, we train a series of VL models that autoregressively generate text from arbitrary combinations of visual and textual input. The pretraining is entirely end-to-end using a single language modeling objective, simplifying o…
Addressing Hallucinations in Language Models with Knowledge Graph Embeddings as an Additional Modality
Institute for Clarity in Documentation（文献清晰研究所）；Inria Paris-Rocquencourt（法国国家信息与自动化研究所巴黎-罗康校区）；Rajiv Gandhi University（拉吉夫·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕尔默研究实验室）
Semantic Relation-Enhanced CLIP Adapter for Domain Adaptive Zero-Shot Learning
School of Computer Science and Technology, East China Normal University, China（东华大学计算机科学与技术学院）；College of Computer Science and Technology, Ocean University of China, China（中国海洋大学计算机科学与技术学院）
How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not
Sapienza University of Rome（罗马萨皮恩扎大学）；Roma Tre University（罗马三大学）；TranslatedItaly（译意大利）；Pi School（皮学校）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）；AGH University of Krakow（克拉科夫AGH大学）；ACC Cyfronet AGH
BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；OpenDataLab, Shanghai Artificial Intelligence Laboratory（上海人工智能实验室 OpenDataLab）；Zhejiang University（浙江大学）；Shanghai Innovation Institute（上海创新研究院）；East China Normal University（华东师范大学）；Zhongguancun Academy（中关村学院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
Fine-Grained VLM Fine-tuning via Latent Hierarchical Adapter Learning
Information Materials and Intelligent Sensing Laboratory of Anhui Province（安徽省信息材料与智能感知实验室）；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）
PE-CLIP: A Parameter-Efficient Fine-Tuning of Vision Language Models for Dynamic Facial Expression Recognition
Univ. BTU Cottbus-Senftenberg（科特布斯-森夫滕堡应用科学大学）；Sorbonne University Abu Dhabi（索邦大学阿布扎比分校）；Univ. Polytechnique Hauts-de-France（上法兰西理工大学）；Centrale Lille（里尔中央理工学院）；CNRS（法国国家科学研究中心）
LangBridge: Interpreting Image as a Combination of Language Embeddings
Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）；Peking University（北京大学）；PengCheng Laboratory（鹏城实验室）；Chongqing University（重庆大学）
Omni-Safety under Cross-Modality Conflict: Vulnerabilities, Dynamics Mechanisms and Efficient Alignment
Nanyang Technological University（南洋理工大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Tsinghua University（清华大学）；Fudan University（复旦大学）；University of Science and Technology of China（中国科学技术大学）；Renmin University of China（中国人民大学）
The task of RGBT tracking aims to take the complementary advantages from visible spectrum and thermal infrared data to achieve robust visual tracking, and receives more and more attention in recent years. Existing works focus on modality-specific information integration by introducing modality weights to achieve adaptive fusion or learning robust feature representations of different modalities. Although these methods could effectively deploy the modality-specific properties, they ignore the potential values of modality-shared cues as well as instance-aware information, which are crucial for ef…
RGBT tracking has attracted increasing attention since RGB and thermal infrared data have strong complementary advantages, which could make trackers all-day and all-weather work. However, how to effectively represent RGBT data for visual tracking remains unstudied well. Existing works usually focus on extracting modality-shared or modality-specific information, but the potentials of these two cues are not well explored and exploited in RGBT tracking. In this paper, we propose a novel multi-adapter network to jointly perform modality-shared, modality-specific and instance-aware target represent…
