Recent years have witnessed the resurgence of knowledge engineering which is featured by the fast growth of knowledge graphs. However, most of existing knowledge graphs are represented with pure symbols, which hurts the machine's capability to understand the real world. The multi-modalization of knowledge graphs is an inevitable key step towards the realization of human-level machine intelligence. The results of this endeavor are Multi-modal Knowledge Graphs (MMKGs). In this survey on MMKGs constructed by texts and images, we first give definitions of MMKGs, followed with the preliminaries on multi-modal tasks and techniques. We then systematically review the challenges, progresses and opportunities on the construction and application of MMKGs respectively, with detailed analyses of the strength and weakness of different solutions. We finalize this survey with open research problems relevant to MMKGs.
NativE: Multi-modal Knowledge Graph Completion in the Wild
Zhejiang University（浙江大学）；Ant Group（蚂蚁集团）；Zhejiang University-Ant Group Joint Laboratory of Knowledge Graph（浙江大学-蚂蚁集团知识图谱联合实验室）；Alibaba-Zhejiang University Joint Institute of Frontier Technology（阿里巴巴-浙江大学前沿技术联合研究院）
Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion
Zhejiang University（浙江大学）；Ant Group（蚂蚁集团）；Alibaba-Zhejiang University Joint Institute of Frontier Technology（阿里巴巴-浙江大学前沿技术联合研究中心）；Zhejiang University-Ant Group Joint Laboratory of Knowledge Graph（浙江大学-蚂蚁集团知识图谱联合实验室）
Integrating Knowledge Graph embedding and pretrained Language Models in Hypercomplex Spaces
University of Stuttgart（斯图加特大学）；University of Bonn（波恩大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；TU Dresden（德累斯顿工业大学）；Amazon（亚马逊）；University of Southampton（南安普顿大学）
Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey
Zhejiang University（浙江大学）；Hangzhou Dianzi University（杭州电子科技大学）；The University of Manchester（曼彻斯特大学）；University of Oxford（牛津大学）；Southeast University（东南大学）；Purdue University（普渡大学）；The University of Edinburgh（爱丁堡大学）
Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；ZJU-Ant Group Joint Lab of Knowledge Graph（浙江大学-蚂蚁集团知识图谱联合实验室）；Ant Group（蚂蚁集团）；School of Software Technology, Zhejiang University（浙江大学软件学院）；Zhejiang Key Laboratory of Big Data Intelligent Computing（浙江省大数据智能计算重点实验室）
A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Innovation Center of Yangtze River Delta, Zhejiang University（浙江大学长三角创新中心）
Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluation
College of Electronics and Information Engineering, Sichuan University（四川大学电子信息工程学院）；School of Computer Science, University of Adelaide（阿德莱德大学计算机科学学院）；School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）
From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；School of Computer Science and Engineering, Tianjin University of Technology（天津理工大学计算机科学与工程学院）；Peng Cheng Laboratory（鹏城实验室）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Peng Cheng Laboratory（鹏城实验室）
Bridging Modalities: Knowledge Distillation and Masked Training for Translating Multi-Modal Emotion Recognition to Uni-Modal, Speech-Only Emotion Recognition
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Software, Beihang University（北京航空航天大学软件学院）；Shandong Inspur Intelligent Production Technology Co., Ltd（山东浪潮智能生产技术有限公司）
Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards
University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）；NVIDIA（英伟达公司）；Penn State University（宾夕法尼亚州立大学）；Columbia University（哥伦比亚大学）；University of Missouri-Kansas City（密苏里大学堪萨斯分校）；Florida State University（佛罗里达州立大学）；Auburn University（奥本大学）
Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm
School of Information and Communications Technology, Hanoi University of Science and Technology, Hanoi, Vietnam（信息与通信技术学院，河内科学技术大学，越南河内）；School of Electrical Engineering and Computer Science, Washington State University, Pullman, US（电气工程与计算机科学学院，华盛顿州立大学，华盛顿州普尔曼）
MolKD: Distilling Cross-Modal Knowledge in Chemical Reactions for Molecular Property Prediction
Institute for Interdisciplinary Information Sciences (IIIS), Tsinghua University（清华大学交叉信息研究院）；Zhejiang Lab（之江实验室）；The Chinese University of Hong Kong（香港中文大学）
Harmony: A Unified Framework for Modality Incremental Learning
Peng Cheng Laboratory（鹏城实验室）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）
