Structure Guided Multi-modal Pre-trained Transformer for Knowledge Graph Reasoning
National University of Defense Technology(国防科技大学)
Ke Liang, Sihang Zhou, Yue Liu, Lingyuan Meng, Meng Liu, Xinwang Liu
备注:
This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessed
Multimodal knowledge graphs (MKGs), which intuitively organize information in various modalities, can benefit multiple practical downstream tasks, such as recommendation systems, and visual question answering. However, most MKGs are still far from complete, which motivates the flourishing of MKG reasoning models. Recently, with the development of general artificial architectures, the pretrained transformer models have drawn increasing attention, especially for multimodal scenarios. However, the research of multimodal pretrained transformer (MPT) for knowledge graph reasoning (KGR) is still at an early stage. As the biggest difference between MKG and other multimodal data, the rich structural information underlying the MKG still cannot be fully leveraged in existing MPT models. Most of them only utilize the graph structure as a retrieval map for matching images and texts connected with the same entity. This manner hinders their reasoning performances. To this end, we propose the graph Structure Guided Multimodal Pretrained Transformer for knowledge graph reasoning, termed SGMPT. Specifically, the graph structure encoder is adopted for structural feature encoding. Then, a structure-guided fusion module with two different strategies, i.e., weighted summation and alignment constraint, is first designed to inject the structural information into both the textual and visual features. To the best of our knowledge, SGMPT is the first MPT model for multimodal KGR, which mines the structural information underlying the knowledge graph. Extensive experiments on FB15k-237-IMG and WN18-IMG, demonstrate that our SGMPT outperforms existing state-of-the-art models, and prove the effectiveness of the designed strategies.
Guangdong University of Technology(广东工业大学);University of Copenhagen(哥本哈根大学);Nanjing University(南京大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Tencent(腾讯);Arizona State University(亚利桑那州立大学)
MPFlow: Multi-modal Posterior-Guided Flow Matching for Zero-Shot MRI Reconstruction
1 Hawkes Institute, UCL \, 2 Dept. of Medical Physics;Biomedical Engineering, UCL 3 Dept. of Computer Science, UCL 4 Centre for AI, DS\&AI, AstraZeneca, UK
NEAT: Neighborhood-Guided, Efficient, Autoregressive Set Transformer for 3D Molecular Generation
Department of Pharmaceutical Sciences, University of Vienna(维也纳大学药学系);Christian Doppler Laboratory for Molecular Informatics in the Biosciences(生物医学分子信息学克里斯蒂安·多普勒实验室);Vienna Doctoral School of Pharmaceutical, Nutritional and Sport Sciences(维也纳药学、营养学和运动科学博士学院)
OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉科学学院);Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所中文信息处理实验室);University of Limerick(利默里克大学);CUHK, Shenzhen(香港中文大学(深圳))
CorMulT: A Semi-supervised Modality Correlation-aware Multimodal Transformer for Sentiment Analysis
Information Networking Institute at Carnegie Mellon University (CMU)(卡内基梅隆大学信息网络研究所);School of Computer Science and Technology at Tongji University(同济大学计算机科学与技术学院)
Future-Guided Incremental Transformer for Simultaneous Translation
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)
Hybrid Explanation-Guided Learning for Transformer-Based Chest X-Ray Diagnosis
ARTORG Center for Biomedical Engineering Research, University of Bern(ARTORG生物医学工程研究中心,伯恩大学);Inselspital (Bern University Hospital)(Inselspital(伯恩大学医院));Insel Gruppe Bern Universitätsinstitut für Diagnostische, Interventionelle und Pädiatrische Radiologie(Bern大学诊断、介入和儿科放射学研究所);Department of Radiation Oncology, Inselspital, Bern University Hospital(放射肿瘤科,Inselspital,伯恩大学医院)