Large Language Models (LLMs) have shown remarkable success, and their multimodal expansions (MLLMs) further unlock capabilities spanning images, videos, and other modalities beyond text. However, despite this shift, prompt optimization approaches, designed to reduce the burden of manual prompt crafting while maximizing performance, remain confined to text, ultimately limiting the full potential of MLLMs. Motivated by this gap, we introduce the new problem of multimodal prompt optimization, which expands the prior definition of prompt optimization to the multimodal space defined by the pairs of textual and non-textual prompts. To tackle this problem, we then propose the Multimodal Prompt Optimizer (MPO), a unified framework that not only performs the joint optimization of multimodal prompts through alignment-preserving updates but also guides the selection process of candidate prompts by leveraging earlier evaluations as priors in a Bayesian-based selection strategy. Through extensive experiments across diverse modalities that go beyond text, such as images, videos, and even molecules, we demonstrate that MPO outperforms leading text-only optimization methods, establishing multimodal prompt optimization as a crucial step to realizing the potential of MLLMs.
When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMs
Peking University（北京大学）；Provable Responsible AI and Data Analytics (PRADA) Lab（可证明负责任的人工智能和数据分析实验室）；King Abdullah University of Science and Technology（卡布斯大学）；South China University of Technology（华南理工大学）；Tsinghua University（清华大学）；University of Georgia（佐治亚大学）；MBZUAI（马克斯·普朗克人工智能研究所）
Model Composition for Multimodal Large Language Models
Tsinghua University（清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Institute of Intelligent Computing, Alibaba Group（阿里巴巴集团智能计算研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
LLMs Can Evolve Continually on Modality for X-Modal Reasoning
Dalian University of Technology（大连理工大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology（香港科技大学）
HKUST(GZ)（香港科技大学（广州））；CSE, HKUST（香港科技大学计算机科学与工程系）；Xi’an Jiaotong University（西安交通大学）；University of Pisa, IT（比萨大学）；University of Trento, IT（特伦特大学）；Nagoya University（名古屋大学）；China University of Mining & Technology, Beijing（中国矿业大学（北京））；Tongji University（同济大学）；SPIC Energy Science and Technology Research Institute（SPIC能源科学与技术研究院）；Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）；College of Computing & Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）
Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Loop Area Institute（深圳河套学院）；National Health Data Institute (Shenzhen)（国家健康数据研究院（深圳））
Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMs
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies（广东省新型安全智能技术重点实验室）
M$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning
Rochester Institute of Technology（罗切斯特理工学院）；U.S. Naval Research Laboratory（美国海军研究实验室）；Harvard Medical School（哈佛医学院）；ByteDance（字节跳动）；Meta AI；Meituan（美团）；University of Science and Technology of China（中国科学技术大学）；Shanghai Academy of AI for Science（上海人工智能科学研究院）；Fudan University（复旦大学）；University of Missouri - Kansas City（密苏里大学堪萨斯城分校）；University of California - Davis（加利福尼亚大学戴维斯分校）
Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting
School of Computer Science, South China Normal University（华南师范大学计算机科学学院）；School of Information Technology in Education, South China Normal University（华南师范大学教育信息技术学院）
How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model
College of Computer, National University of Defense Technology（国防科技大学计算机学院）；School of Computer and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）
Find Them All: Unveiling MLLMs for Versatile Person Re-identification
School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；Institute of Image Communication and Information Processing, Shanghai Jiao Tong University（上海交通大学图像通信与信息处理研究所）；Macao Polytechnic University（澳门 polytechnic 大学）；Shanghai AI Laboratory（上海人工智能实验室）
Surveying the MLLM Landscape: A Meta-Review of Current Surveys
Georgia Institute of Technology（佐治亚理工学院）；Indiana University（印第安纳大学）；Purdue University（普渡大学）；Emory University（埃默里大学）；Sichuan University（四川大学）；Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学）；AppCubic；Kyoto University（京都大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Rutgers University（罗格斯大学）；National Taiwan Normal University（台湾师范大学）
From Consistency to Complementarity: Aligned and Disentangled Multi-modal Learning for Time Series Understanding and Reasoning
The Hong Kong University of Science（香港科学与技术大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
WebRISE: Requirement-Induced State Evaluation for MLLM-Generated Web Artifacts
Tsinghua University（清华大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；East China Normal University（华东师范大学）；Tongji University（同济大学）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）
Why Text Prevails: Vision May Undermine Multimodal Medical Decision Making
University of Texas Rio Grande Valley（德克萨斯大学里奥格兰德谷大学）；University of Pittsburgh（匹兹堡大学）；NC State University（北卡罗来纳州立大学）；University of Washington（华盛顿大学）；University of Maryland（马里兰大学）
OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks
Chinese University of Hong Kong（香港中文大学）；Sun Yat-sen Memorial Hospital, Sun Yat-sen University（中山大学孙逸仙纪念医院）；Southern Medical University（南方医科大学）；Zhongshan Hospital, Fudan University（复旦大学中山医院）；Department of Neurosurgery, Prince of Wales Hospital（威尔士王室医院神经外科部）
TCAP: Tri-Component Attention Profiling for Unsupervised Backdoor Detection in MLLM Fine-Tuning
School of Control Science and Engineering, Shandong University, Jinan, China（控制科学与工程学院，山东大学，济南，中国）；Key Laboratory of Industrial Intelligent Systems, Shandong Province, China（山东省工业智能系统重点实验室，中国）
A Survey of Multimodal Large Language Model from A Data-centric Perspective
Hong Kong University of Science and Technology（香港科技大学）；Apple（苹果公司）；Peking University（北京大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Science and Technology of China（中国科学技术大学）
Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Models
School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机科学与信息工程学院）；Tsinghua University（清华大学）；Institute of automation, Chinese academy of science（中国科学院自动化研究所）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）
