Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
Symbiosis Institute of Technology (SIT), Symbiosis International (Deemed University)(共生国际(视为大学)共生技术学院);Symbiosis Centre for Applied Artificial Intelligence, Symbiosis Institute of Technology (SIT), Symbiosis International (Deemed University)(共生国际(视为大学)共生技术学院共生应用人工智能中心);University of Winnipeg(温尼伯大学)
Multimodal deep learning systems which employ multiple modalities like text, image, audio, video, etc., are showing better performance in comparison with individual modalities (i.e., unimodal) systems. Multimodal machine learning involves multiple aspects: representation, translation, alignment, fusion, and co-learning. In the current state of multimodal machine learning, the assumptions are that all modalities are present, aligned, and noiseless during training and testing time. However, in real-world tasks, typically, it is observed that one or more modalities are missing, noisy, lacking annotated data, have unreliable labels, and are scarce in training or testing and or both. This challenge is addressed by a learning paradigm called multimodal co-learning. The modeling of a (resource-poor) modality is aided by exploiting knowledge from another (resource-rich) modality using transfer of knowledge between modalities, including their representations and predictive models. Co-learning being an emerging area, there are no dedicated reviews explicitly focusing on all challenges addressed by co-learning. To that end, in this work, we provide a comprehensive survey on the emerging area of multimodal co-learning that has not been explored in its entirety yet. We review implementations that overcome one or more co-learning challenges without explicitly considering them as co-learning challenges. We present the comprehensive taxonomy of multimodal co-learning based on the challenges addressed by co-learning and associated implementations. The various techniques employed to include the latest ones are reviewed along with some of the applications and datasets. Our final goal is to discuss challenges and perspectives along with the important ideas and directions for future work that we hope to be beneficial for the entire research community focusing on this exciting domain.
ADS-Edit: A Multimodal Knowledge Editing Dataset for Autonomous Driving Systems
Zhejiang University,\ University - Ant \ Joint Laboratory of Knowledge Graph Hangzhou China;Zhejiang University,\ University - Ant \ Joint Laboratory of Knowledge Graph
From Classical Machine Learning to Emerging Foundation Models: Review on Multimodal Data Integration for Cancer Research
Department of Imaging Physics, The University of Texas MD Anderson Cancer Center(影像物理系,德克萨斯大学MD安德森癌症中心);Department of Thoracic/Head and Neck Medical Oncology, The University of Texas MD Anderson Cancer Center(胸腔/头颈医学肿瘤科,德克萨斯大学MD安德森癌症中心);Department of Thoracic Radiation Oncology, The University of Texas MD Anderson Cancer Center(胸腔放射肿瘤科,德克萨斯大学MD安德森癌症中心);Department of Translational Molecular Pathology, The University of Texas MD Anderson Cancer Center(转化分子病理学系,德克萨斯大学MD安德森癌症中心);Department of Thoracic Imaging, The University of Texas MD Anderson Cancer Center(胸腔影像科,德克萨斯大学MD安德森癌症中心)
Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions
School of Biomedical Engineering, Tsinghua University(清华大学生物医学工程学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Hepato-Pancreato-Biliary Center, Beijing Tsinghua Changgung Hospital, School of Clinical Medicine, Tsinghua University(清华大学临床医学院北京清华长庚医院肝胆胰中心);School of Computer and Communication Engineering, University of Science and Technology Beijing(北京科技大学计算机与通信工程学院);College of Electronic Information Engineering, Tongji University(同济大学电子与信息工程学院)
Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models
Technical University of Munich(慕尼黑工业大学);Hong Kong Baptist University(香港浸会大学);Aalto University(阿尔托大学);University of Oulu(奥卢大学);University of Bonn(波恩大学);EPFL(洛桑联邦理工学院)
From Efficient Multimodal Models to World Models: A Survey
Fudan University(复旦大学);Academy for Engineering and Technology, Fudan University(复旦大学工程与技术学院);Shanghai Engineering Research Center of AI and Robotics(上海人工智能与机器人工程研究中心);Engineering Research Center of AI and Robotics, Ministry of Education(教育部人工智能与机器人工程研究中心);Shanghai Key Lab of Intelligent Information Processing(上海智能信息处理重点实验室);School of Computer Science, Fudan University(复旦大学计算机科学技术学院)
Multimodal Fusion on Low-quality Data: A Comprehensive Survey
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局高性能计算研究所);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);College of Computer Science, Sichuan University(四川大学计算机学院);School of Computer Science and Technology, Xidian University(西安电子科技大学计算机科学与技术学院);School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)计算机科学与技术学院)