Multi-modal Large Language Models (MLLMs) have recently exhibited impressive general-purpose capabilities by leveraging vision foundation models to encode the core concepts of images into representations. These are then combined with instructions and processed by the language model to generate high-quality responses. Despite significant progress in enhancing the language component, challenges persist in optimally fusing visual encodings within the language model for task-specific adaptability. Recent research has focused on improving this fusion through modality adaptation modules but at the cost of significantly increased model complexity and training data needs. In this paper, we propose EMMA (Efficient Multi-Modal Adaptation), a lightweight cross-modality module designed to efficiently fuse visual and textual encodings, generating instruction-aware visual representations for the language model. Our key contributions include: (1) an efficient early fusion mechanism that integrates vision and language representations with minimal added parameters (less than 0.2% increase in model size), (2) an in-depth interpretability analysis that sheds light on the internal mechanisms of the proposed method; (3) comprehensive experiments that demonstrate notable improvements on both specialized and general benchmarks for MLLMs. Empirical results show that EMMA boosts performance across multiple tasks by up to 9.3% while significantly improving robustness against hallucinations. Our code is available at https://github.com/SaraGhazanfari/EMMA
EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment
Peng Cheng Laboratory(鹏城实验室);University of Chinese Academy of Sciences(中国科学院大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Sun Yat-sen University(中山大学)
Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark
University of Electronic Science and Technology of China(电子科技大学);Sun Yat-sen University(中山大学);University of Washington(华盛顿大学);Microsoft(微软公司);The Chinese University of Hong Kong(香港中文大学)
本文提出覆盖数学、物理、化学和编程的 EMMA 多模态推理基准,揭示先进 MLLMs 在复杂跨模态、多步推理上仍存在明显不足。
Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld
Southern University of Science and Technology(南方科技大学);JD Explore Academy(京东探索研究院);University of Technology Sydney(悉尼科技大学);University of Maryland, College Park(马里兰大学学院公园分校);Yunnan University(云南大学)
IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment
Media Integration and Communication Center (MICC), University of Florence, Italy(意大利佛罗伦萨大学媒体集成与通信中心);Department of Computer Science, Universitat Autònoma de Barcelona, Spain(西班牙巴塞罗那自治大学计算机科学系);Computer Vision Center, Barcelona, Spain(西班牙巴塞罗那计算机视觉中心);IDEAS Research Institute, Warsaw, Poland(波兰华沙IDEAS研究所)
Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
ASAP: Advancing Semantic Alignment Promotes Multi-Modal Manipulation Detecting and Grounding
School of Computer Science and Information Engineering, Hefei University of Technology, China(计算机科学与信息工程学院,合肥工业大学,中国);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, China(人工智能研究所,合肥综合性国家科学中心,中国)
Multi-modal Semantic Understanding with Contrastive Cross-modal Feature Alignment
Peking University(北京大学);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);School of Computer Science, Peking University(北京大学计算机学院)
Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning
School of Automation, Northwestern Polytechnical University(西北工业大学自动化学院);City University of Hong Kong(香港城市大学);The University of Texas at Arlington(德克萨斯大学阿灵顿分校);School of Biomedical Engineering, ShanghaiTech University(上海科技大学生物医学工程学院);Shanghai United Imaging Intelligence Co., Ltd.(上海联影智能医疗科技有限公司);Shanghai Clinical Research and Trial Center(上海临床研究与试验中心);University of Georgia(佐治亚大学);Massachusetts General Hospital(麻省总医院)
Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment
Zhejiang University(浙江大学);The University of Manchester(曼彻斯特大学);University of Oxford(牛津大学);The University of Edinburgh(爱丁堡大学);Tsinghua University(清华大学);Donghai laboratory(东海实验室)
AlignFreeNet: Is Cross-Modal Pre-Alignment Necessary? An End-to-End Alignment-Free Lightweight Network for Visible-Infrared Object Detection
School of Computer Science, Jiangsu University of Technology(江苏科技大学计算机科学学院);School of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics(南京航空航天大学计算机科学与技术学院);School of Science and Technology, Hong Kong Metropolitan University(香港都会大学科技学院);School of Data Science, Lingnan University(岭南大学数据科学学院);Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院)