Converting different modalities into generalized text, which then serves as input prompts for large language models (LLMs), is a common approach for aligning multimodal models, particularly when pairwise data is limited. Text-centric alignment method leverages the unique properties of text as a modality space, transforming diverse inputs into a unified textual representation, thereby enabling downstream models to effectively interpret various modal inputs. This study evaluates the quality and robustness of multimodal representations in the face of noise imperfections, dynamic input order permutations, and missing modalities, revealing that current text-centric alignment methods can compromise downstream robustness. To address this issue, we propose a new text-centric adversarial training approach that significantly enhances robustness compared to traditional robust training methods and pre-trained multimodal foundation models. Our findings underscore the potential of this approach to improve the robustness and adaptability of multimodal representations, offering a promising solution for dynamic and real-world applications.
Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Hebei University of Science and Technology（河北科技大学）
Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention
University of California, Riverside（加州大学河滨分校）；University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）；Qatar Computing Research Institute (QCRI)（卡塔尔计算研究所）
PAL: Prompting Analytic Learning with Missing Modality for Multi-Modal Class-Incremental Learning
National University of Singapore（新加坡国立大学）；National University of Defense Technology（国防科技大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局信息通信研究所）；Saw Swee Hock School of Public Health, National University of Singapore（新加坡国立大学苏瑞福公共卫生学院）；Shien-Ming Wu School of Intelligent Engineering, South China University of Technology（华南理工大学伍舜德智能工程学院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
Multimodal-Guided Dynamic Dataset Pruning for Robust and Efficient Data-Centric Learning
National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）
Buffer replay enhances the robustness of multimodal learning under missing-modality
College of Computer Science and Electronic Engineering Hunan University（湖南大学计算机科学与电子工程学院）；School of Computer Science Central South University（中南大学计算机学院）
Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective
University of Science and Technology of China（中国科学技术大学）；Eindhoven University of Technology（埃因霍温理工大学）；University of Exeter（埃克塞特大学）；Michigan State University（密歇根州立大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；MIT（麻省理工学院）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Harvard University（哈佛大学）
Jailbreaking Vision-Language Models Through the Visual Modality
Warsaw University of Technology（华沙理工大学）；NASK National Research Institute（波兰国家研究研究院）；University of Liverpool（利物浦大学）；Indian Institute of Technology Roorkee（印度理工学院拉胡尔分校）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
