Machine Learning (ML) is continuously permeating a growing amount of application domains. Generative AI such as Large Language Models (LLMs) also sees broad adoption to process multi-modal data such as text, images, audio, and video. While the trend is to use ever-larger datasets for training, managing this data efficiently has become a significant practical challenge in the industry-double as much data is certainly not double as good. Rather the opposite is important since getting an understanding of the inherent quality and diversity of the underlying data lakes is a growing challenge for application-specific ML as well as for fine-tuning foundation models. Furthermore, information retrieval (IR) from expanding data lakes is complicated by the temporal dimension inherent in time-series data which must be considered to determine its semantic value. This study focuses on the different semantic-aware techniques to extract embeddings from mono-modal, multi-modal, and cross-modal data to enhance IR capabilities in a growing data lake. Articles were collected to summarize information about the state-of-the-art techniques focusing on applications of embedding for three different categories of data modalities.
Review of multimodal machine learning approaches in healthcare
Oxford Internet Institute, University of Oxford(牛津大学互联网研究所);Institute for People-Centred AI, University of Surrey(萨里大学以人为本人工智能研究所);Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London(伦敦大学学院惠康/英国工程与物理科学研究委员会介入与外科科学中心)
AI-based Automatic Segmentation of Prostate on Multi-modality Images: A Review
Soochow University(苏州大学);Peking University First Hospital(北京大学第一医院);Duke Kunshan University(昆山杜克大学);Swinburne University of Technology(斯威本科技大学);The Hong Kong Polytechnic University(香港理工大学);UT Southwestern Medical Center(德克萨斯大学西南医学中心)
Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
Symbiosis Institute of Technology (SIT), Symbiosis International (Deemed University)(共生国际(视为大学)共生技术学院);Symbiosis Centre for Applied Artificial Intelligence, Symbiosis Institute of Technology (SIT), Symbiosis International (Deemed University)(共生国际(视为大学)共生技术学院共生应用人工智能中心);University of Winnipeg(温尼伯大学)
Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyond
National Heart and Lung Institute, Imperial College London(帝国理工学院国家心肺研究所);Royal Brompton Hospital(皇家布朗普顿医院);Imperial Institute of Advanced Technology(帝国理工高等研究院);Hangzhou Ocean’s Smart Boya Co., Ltd(杭州海洋智能博雅有限公司);University of California, San Diego(加州大学圣地亚哥分校);Shenzhen Second People’s Hospital(深圳市第二人民医院)
Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey
Zhejiang University(浙江大学);Hangzhou Dianzi University(杭州电子科技大学);The University of Manchester(曼彻斯特大学);University of Oxford(牛津大学);Southeast University(东南大学);Purdue University(普渡大学);The University of Edinburgh(爱丁堡大学)
Representation Learning with Mutual Influence of Modalities for Node Classification in Multi-Modal Heterogeneous Networks
Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学);School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院);Binzhou Institute of Technology(滨州职业技术学院)
From Pixels to Prose: Advancing Multi-Modal Language Models for Remote Sensing
Simon Fraser University(西蒙弗雷泽大学);University of Minnesota - Twin Cities(明尼苏达大学双城分校);Kyoto University(京都大学);Georgia Institute of Technology(佐治亚理工学院);National Taiwan Normal University(台湾师范大学);Purdue University(普渡大学);Emory University(埃默里大学);The University of Texas at Dallas(德克萨斯大学达拉斯分校)