Although In-Context Learning (ICL) brings remarkable performance gains to Large Language Models (LLMs), the improvements remain lower than fine-tuning on downstream tasks. This paper introduces Multi-Modal In-Context Tuning (MMICT), a novel multi-modal fine-tuning paradigm that boosts multi-modal fine-tuning by fully leveraging the promising ICL capability of multi-modal LLMs (MM-LLMs). We propose the Multi-Modal Hub (M-Hub), a unified module that captures various multi-modal features according to different inputs and objectives. Based on M-Hub, MMICT enables MM-LLMs to learn from in-context visual-guided textual features and subsequently generate outputs conditioned on the textual-guided visual features. Moreover, leveraging the flexibility of M-Hub, we design a variety of in-context demonstrations. Extensive experiments on a diverse range of downstream multi-modal tasks demonstrate that MMICT significantly outperforms traditional fine-tuning strategy and the vanilla ICT method that directly takes the concatenation of all information from different modalities as input. Our implementation is available at: https://github.com/KDEGroup/MMICT.
Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Intelligence Science and Technology, University of Science and Technology Beijing（北京科技大学智能科学与技术学院）
Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting
School of Computer Science, South China Normal University（华南师范大学计算机科学学院）；School of Information Technology in Education, South China Normal University（华南师范大学教育信息技术学院）
TinyAlign: Boosting Lightweight Vision-Language Models by Mitigating Modal Alignment Bottlenecks
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing（北京未来区块链与隐私计算先进创新中心）；Beihang University（北京航空航天大学）；Hangzhou International Innovation Institute（杭州国际创新研究院）；Xreal；Renmin University（中国人民大学）；Peking University（北京大学）；Beijing Academy of Blockchain and Edge Computing (BABEC)（北京区块链与边缘计算研究院）
Boosting the Power of Small Multimodal Reasoning Models to Match Larger Models with Self-Consistency Training
Zhejiang University（浙江大学）；Westlake University（西湖大学）；Shenyang Institute of Computing Technology, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Advanced Technology, Westlake Institute for Advanced Study（西湖高等研究院先进技术研究院）
CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance
Kyung Hee University（庆熙大学）；The University of Danang—Vietnam-Korea University of Information and Communication Technology（岘港大学—越韩信息与通信技术大学）；Digital Science and Technology Institute（数字科学技术研究院）
Korea Advanced Institute of Science and Technology / Korea, Republic of（韩国科学技术院）；University of Illinois in Urbana-Champaign / United States of America（伊利诺伊大学厄巴纳-香槟分校）；Korea University / Korea, Republic of（韩国大学）
ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Models
School of Computer Science and Engineering, Central South University（计算机科学与工程学院，中南大学）；Research Center for SCIR, Harbin Institute of Technology（SCIR研究中心，哈尔滨工业大学）；NExT Research Center, National University of Singapore（NExT研究中心，新加坡国立大学）
Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
ReconBoost: Boosting Can Achieve Modality Reconcilement
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Key Laboratory of Big Data Mining and Knowledge Management, Chinese Academy of Sciences（中国科学院大数据挖掘与知识管理重点实验室）
Boosting Multi-modal Model Performance with Adaptive Gradient Modulation
ShanghaiTech University（上海科技大学）；Shanghai Innovation Center for Processor Technologies（上海处理器技术创新中心）；Shanghai Center for Brain Science and Brain-Inspired Technology（上海脑科学与类脑研究中心）；University of Science and Technology of China（中国科学技术大学）
CMRxRecon2024: A Multi-Modality, Multi-View K-Space Dataset Boosting Universal Machine Learning for Accelerated Cardiac MRI
Xiamen University（厦门大学）；Imperial College London（帝国理工学院）；Royal Brompton Hospital（皇家布朗普顿医院）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Fudan University（复旦大学）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）；University of Shanghai for Science and Technology（上海理工大学）；GE Healthcare（通用电气医疗集团）；Siemens Healthineers Ltd.（西门子医疗有限公司）；The Hong Kong Polytechnic University（香港理工大学）；Pontificia Universidad Católica de Chile（智利天主教大学）
