Existing multimodal sentiment analysis tasks are highly rely on the assumption that the training and test sets are complete multimodal data, while this assumption can be difficult to hold: the multimodal data are often incomplete in real-world scenarios. Therefore, a robust multimodal model in scenarios with randomly missing modalities is highly preferred. Recently, CLIP-based multimodal foundational models have demonstrated impressive performance on numerous multimodal tasks by learning the aligned cross-modal semantics of image and text pairs, but the multimodal foundational models are also unable to directly address scenarios involving modality absence. To alleviate this issue, we propose a simple and effective framework, namely TRML, Toward Robust Multimodal Learning using Multimodal Foundational Models. TRML employs generated virtual modalities to replace missing modalities, and aligns the semantic spaces between the generated and missing modalities. Concretely, we design a missing modality inference module to generate virtual modaliites and replace missing modalities. We also design a semantic matching learning module to align semantic spaces generated and missing modalities. Under the prompt of complete modality, our model captures the semantics of missing modalities by leveraging the aligned cross-modal semantic space. Experiments demonstrate the superiority of our approach on three multimodal sentiment analysis benchmark datasets, CMU-MOSI, CMU-MOSEI, and MELD.
Analyzing Modality Robustness in Multimodal Sentiment Analysis
National University of Singapore(新加坡国立大学);Beijing University of Posts and Telecommunications(北京邮电大学);Singapore University of Technology and Design(新加坡科技设计大学)
A Foundational Multimodal Vision Language AI Assistant for Human Pathology
Brigham and Women’s Hospital(布莱根妇女医院);Harvard Medical School(哈佛医学院);Massachusetts General Hospital(麻省总医院);Broad Institute of Harvard and MIT(哈佛-麻省理工博德研究所);Dana-Farber Cancer Institute(达纳-法伯癌症研究所);Massachusetts Institute of Technology (MIT)(麻省理工学院);Harvard John A. Paulson School of Engineering And Applied Sciences(哈佛大学约翰·A·保尔森工程与应用科学学院);Harvard University(哈佛大学);Wexner Medical Center, Ohio State University(俄亥俄州立大学韦克斯纳医学中心);Ohio State University(俄亥俄州立大学);Harvard Data Science Initiative(哈佛大学数据科学计划)
Efficient Domain Adaptation of Multimodal Embeddings using Constrastive Learning
Operations Research Center, Massachusetts Institute of Technology(麻省理工学院运营研究中心);Sloan School of Management, Massachusetts Institute of Technology(麻省理工学院斯隆管理学院)
A Survey of Multimodal Ophthalmic Diagnostics: From Task-Specific Approaches to Foundational Models
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China(深圳大学计算机科学与软件工程学院);Shenzhen Key Laboratory of Visual Object Detection and Recognition, Harbin Institute of Technology, Shenzhen, 518055, China(视觉对象检测与识别深圳重点实验室);Laboratory for Artificial Intelligence in Design, Hong Kong(人工智能设计实验室);School of Artificial Intelligence, Shenzhen University, Shenzhen, China(深圳大学人工智能学院)
LeafNet: A Large-Scale Dataset and Comprehensive Benchmark for Foundational Vision-Language Understanding of Plant Diseases
School of Electrical Engineering, Korea University(韩国大学电气工程学院);Department of Integrative Biology, The University of Texas at Austin(德克萨斯大学奥斯汀分校整合生物学系);Department of Software Engineering, FPT University(FPT大学软件工程系)
PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Models
Peking University(北京大学);Mohamed bin Zayed University of Artificial Intelligence(莫扎伊德大学人工智能学院);National University of Singapore(新加坡国立大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);University of Science and Technology of China(中国科学技术大学);Cornell University(康奈尔大学);Hong Kong Polytechnic University(香港理工大学);City University of Hong Kong(香港城市大学)
Grounding Foundational Vision Models with 3D Human Poses for Robust Action Recognition
Arizona State University(亚利桑那州立大学);Emory University(埃默里大学);University of California, Berkeley(加州大学伯克利分校);Delft University of Technology(代尔夫特理工大学);Algoverse AI Research(Algoverse AI 研究)
Robust Multimodal Learning for Ophthalmic Disease Grading via Disentangled Representation
MBZUAI, United Arab Emirates(MBZUAI,阿联酋);Monash University, Australia(墨尔本大学,澳大利亚);Liverpool University, United Kingdom(利物浦大学,英国);China Unicom (Shanghai) Industrial Internet Co., Ltd., China(中国联合(上海)工业互联网有限公司,中国);Shanghai AI Lab, China(上海人工智能实验室,中国)
RoHyDR: Robust Hybrid Diffusion Recovery for Incomplete Multimodal Emotion Recognition
School of Computer Science and Engineering(计算机科学与工程学院);South China University of Technology(华南理工大学);School of Future Technology(未来技术学院);University of Oxford(牛津大学);Pengcheng Laboratory(鹏城实验室);Department of Computer Science(计算机科学系)
Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion
The Chinese University of Hong Kong(香港中文大学);Imperial College Longdon(伦敦帝国理工学院);Imsight Medical Technology Co. Ltd.(艾迈医疗科技有限公司);The Hong Kong Polytechnic University(香港理工大学);Chinese Academy of Sciences(中国科学院)
BrainMorph: A Foundational Keypoint Model for Robust and Flexible Brain MRI Registration
Department of Radiology, Weill Cornell Medicine, New York City, NY, USA(韦尔·科恩医学中心放射科);School of Electrical and Computer Engineering, Cornell University, Ithaca, NY, USA(康奈尔大学电气与计算机工程学院);Computer Science and Artificial Intelligence Lab at the Massachusetts Institute of Technology, Cambridge, MA, USA(麻省理工学院计算机科学与人工智能实验室)
本文提出基于 KeyMorph 的 BrainMorph 关键点基础模型,利用超十万个 3D 脑 MRI 体积训练,实现鲁棒、可解释、快速可控的多模态成对与组级刚性、仿射及非线性配准。
Training-Free Robust Multimodal Learning via Sample-Wise Jacobian Regularization
Massachusetts Institute of Technology(麻省理工学院);South China University of Technology(华南理工大学);UT-Austin(德克萨斯大学奥斯汀分校);Tsinghua University(清华大学);Shanghai Qizhi Institute(上海期智研究院)
Multimodal-Guided Dynamic Dataset Pruning for Robust and Efficient Data-Centric Learning
National Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家重点实验室,南京大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学)