Multi-modal knowledge graphs (MMKGs) combine different modal data (e.g., text and image) for a comprehensive understanding of entities. Despite the recent progress of large-scale MMKGs, existing MMKGs neglect the multi-aspect nature of entities, limiting the ability to comprehend entities from various perspectives. In this paper, we construct AspectMMKG, the first MMKG with aspect-related images by matching images to different entity aspects. Specifically, we collect aspect-related images from a knowledge base, and further extract aspect-related sentences from the knowledge base as queries to retrieve a large number of aspect-related images via an online image search engine. Finally, AspectMMKG contains 2,380 entities, 18,139 entity aspects, and 645,383 aspect-related images. We demonstrate the usability of AspectMMKG in entity aspect linking (EAL) downstream task and show that previous EAL models achieve a new state-of-the-art performance with the help of AspectMMKG. To facilitate the research on aspect-related MMKG, we further propose an aspect-related image retrieval (AIR) model, that aims to correct and expand aspect-related images in AspectMMKG. We train an AIR model to learn the relationship between entity image and entity aspect-related images by incorporating entity image, aspect, and aspect image information. Experimental results indicate that the AIR model could retrieve suitable images for a given entity w.r.t different aspects.
NativE: Multi-modal Knowledge Graph Completion in the Wild
Zhejiang University(浙江大学);Ant Group(蚂蚁集团);Zhejiang University-Ant Group Joint Laboratory of Knowledge Graph(浙江大学-蚂蚁集团知识图谱联合实验室);Alibaba-Zhejiang University Joint Institute of Frontier Technology(阿里巴巴-浙江大学前沿技术联合研究院)
Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion
Zhejiang University(浙江大学);Ant Group(蚂蚁集团);Alibaba-Zhejiang University Joint Institute of Frontier Technology(阿里巴巴-浙江大学前沿技术联合研究中心);Zhejiang University-Ant Group Joint Laboratory of Knowledge Graph(浙江大学-蚂蚁集团知识图谱联合实验室)
Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation
Zhejiang University(浙江大学);ZJU-Ant Group Joint Lab of Knowledge Graph(浙江大学-蚂蚁集团知识图谱联合实验室);Ant Group(蚂蚁集团);Zhejiang Key Laboratory of Big Data Intelligent Computing(浙江省大数据智能计算重点实验室)
Department of Philosophy, University at Buffalo(布法罗大学哲学系);National Center for Ontological Research(国家本体论研究中心);Institute for Artificial Intelligence and Data Science(人工智能与数据科学研究所);Ontogenesis Solutions, LLC(Ontogenesis Solutions有限责任公司)
Addressing Hallucinations in Language Models with Knowledge Graph Embeddings as an Additional Modality
Institute for Clarity in Documentation(文献清晰研究所);Inria Paris-Rocquencourt(法国国家信息与自动化研究所巴黎-罗康校区);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕尔默研究实验室)
Leveraging Intra-modal and Inter-modal Interaction for Multi-Modal Entity Alignment
School of Computer;Information Technology Shanxi University Taiyuan China;School of Computer Science;Informatics Cardiff University Cardiff UK;ILCC, School of Informatics University of Edinburgh Edinburgh UK;Information Technology Shanxi University;Informatics Cardiff University;ILCC, School of Informatics University of Edinburgh
Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment
Zhejiang University(浙江大学);The University of Manchester(曼彻斯特大学);University of Oxford(牛津大学);The University of Edinburgh(爱丁堡大学);Tsinghua University(清华大学);Donghai laboratory(东海实验室)
Universal Preprocessing Operators for Embedding Knowledge Graphs with Literals
Peoples' Friendship University of Russia (RUDN University)(俄罗斯人民友谊大学(卢蒙巴大学));Joint Institute for Nuclear Research(联合核研究所);Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);University of Skövde(舍夫德大学);University of Mannheim(曼海姆大学)
LoginMEA: Local-to-Global Interaction Network for Multi-modal Entity Alignment
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Baidu Inc.(百度公司)
Mitigating Modality Bias in Multi-modal Entity Alignment from a Causal Perspective
Institute of Information Engineering, Chinese Academy of Sciences(信息工程研究所,中国科学院);School of Cyber Security, University of Chinese Academy of Sciences(网络安全学院,中国科学院大学)
Attribute-Consistent Knowledge Graph Representation Learning for Multi-Modal Entity Alignment
Beihang University(北京航空航天大学);National Computer Network Emergency Response Technical Team/Coordination Center of China(国家计算机网络应急技术处理协调中心);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Zhongguancun Lab(中关村实验室)
IBMEA: Exploring Variational Information Bottleneck for Multi-modal Entity Alignment
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
Characterizing the Entities in Harmful Memes: Who is the Hero, the Villain, the Victim?
Indian Institute of Technology Delhi(德里印度理工学院);Wipro AI Labs(威普罗人工智能实验室);Carnegie Mellon University(卡内基梅隆大学);Indraprastha Institute of Information Technology Delhi(德里英德拉普拉斯塔信息技术学院)
Structural Entities Extraction and Patient Indications Incorporation for Chest X-ray Report Generation
School of Computer Science and Technology, Xidian University(西安电子科技大学计算机科学与技术学院);Xi'an Key Laboratory of Big Data and Intelligent Vision(西安大数据与智能视觉重点实验室);Key Laboratory of Collaborative Intelligence Systems, Ministry of Education, Xidian University(教育部协同智能系统重点实验室);Warren Alpert Medical School, Brown University(布朗大学沃伦·阿尔珀特医学院);School of Electronic Engineering, Xidian University(西安电子科技大学电子工程学院);Department of Radiology and Radiological Sciences, Johns Hopkins University School of Medicine(约翰霍普金斯大学医学院放射科)
ASGEA: Exploiting Logic Rules from Align-Subgraphs for Entity Alignment
School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)
Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);ZJU-Ant Group Joint Lab of Knowledge Graph(浙江大学-蚂蚁集团知识图谱联合实验室);Ant Group(蚂蚁集团);School of Software Technology, Zhejiang University(浙江大学软件学院);Zhejiang Key Laboratory of Big Data Intelligent Computing(浙江省大数据智能计算重点实验室)
BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);University of Science and Technology of China(中国科学技术大学);Huazhong University of Science and Technology(华中科技大学);Microsoft Research(微软研究院)
When Modalities Remember: Continual Learning for Multimodal Knowledge Graphs
School of Computer Science, Peking University(北京大学计算机科学学院);School of Computer Science, Zhejiang University(浙江大学计算机科学学院);School of Cyber Science and Engineering, Wuhan University(武汉大学网络科学与工程学院);School of Information Science and Technology, Tibet University(西藏大学信息科学与技术学院)
Referring Image Segmentation via Cross-Modal Progressive Comprehension
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机学院);Sun Yat-sen University(中山大学);University of Technology Sydney(悉尼科技大学);AI Institute(360人工智能研究院)
Expanding the Boundaries of Vision Prior Knowledge in Multi-modal Large Language Models
University of Chinese Academy of Sciences(中国科学院大学);Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室)
Variational Multi-Modal Hypergraph Attention Network for Multi-Modal Relation Extraction
Beijing University of Posts and Telecommunications(北京邮电大学);Beihang University(北京航空航天大学);CNCERT of China(中国国家互联网应急中心);Zhongguancun Laboratory(中关村实验室);USTC University(中国科学技术大学)