Recently multimodal transformer models have gained popularity because their performance on language and vision tasks suggest they learn rich visual-linguistic representations. Focusing on zero-shot image retrieval tasks, we study three important factors which can impact the quality of learned representations: pretraining data, the attention mechanism, and loss functions. By pretraining models on six datasets, we observe that dataset noise and language similarity to our downstream task are important indicators of model performance. Through architectural analysis, we learn that models with a multimodal attention mechanism can outperform deeper models with modality specific attention mechanisms. Finally, we show that successful contrastive losses used in the self-supervised learning literature do not yield similar performance gains when used in multimodal transformers
Metis-SPECS: Decoupling Multimodal Learning via Self-distilled Preference-based Cold Start
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS）；Meituan（美团）
Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decoupling
Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））；Xiamen University Malaysia（厦门大学马来西亚分校）；Nanyang Technological University（南洋理工大学）；Beijing Academy of Artificial Intelligence, China（北京人工智能研究院）；Peng Cheng Laboratory, China（鹏城实验室）；Shaqra University, Saudi Arabia（沙迦大学）
Correct When Paired, Wrong When Split: Decoupling and Editing Modality-Specific Neurons in MLLMs
School of Information Science and Engineering, Yunnan University（云南大学信息科学与工程学院）；School of Software, Yunnan University（云南大学软件学院）；National University of Singapore（新加坡国立大学）；School of Engineering, Yunnan University（云南大学工程学院）
HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model
School of Advanced Interdisciplinary Sciences, UCAS（UCAS先进交叉科学学院）；MAIS, CASIA（CASIA人工智能研究所）；School of Artificial Intelligence, UCAS（UCAS人工智能学院）；Centre for Artificial Intelligence and Robotics, HKISI-CAS（HKISI-CAS人工智能与机器人中心）；FKLPRIU, School of Computer and Information Engineering, Xiamen University of Technology（厦门理工学院计算机与信息工程学院）
We propose UniT, a Unified Transformer model to simultaneously learn the most prominent tasks across different domains, ranging from object detection to natural language understanding and multimodal reasoning. Based on the transformer encoder-decoder architecture, our UniT model encodes each input modality with an encoder and makes predictions on each task with a shared decoder over the encoded input representations, followed by task-specific output heads. The entire model is jointly trained end-to-end with losses from each task. Compared to previous efforts on multi-task learning with transfo…
Decoupling Semantics and Logic: A Training-Free Coarse-to-Fine Pipeline for Video Retrieval-Augmented Generation
School of Computer Science & Tech, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；School of AI and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）
Decoupled Vision-Language System for Multimodal Understanding and Generation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；Li Auto（理想汽车）
LTCI, Telecom Paris, Institut Polytechnique de Paris（巴黎综合理工学院电信巴黎分校LTCI）；IBM GBS France（IBM GBS法国）
Multimodal sentiment analysis is a trending area of research, and the multimodal fusion is one of its most active topic. Acknowledging humans communicate through a variety of channels (i.e visual, acoustic, linguistic), multimodal systems aim at integrating different unimodal representations into a synthetic one. So far, a consequent effort has been made on developing complex architectures allowing the fusion of these modalities. However, such systems are mainly trained by minimising simple losses such as $L_1$ or cross-entropy. In this work, we investigate unexplored penalties and propose a s…
