Multimodal knowledge graph link prediction aims to improve the accuracy and efficiency of link prediction tasks for multimodal data. However, for complex multimodal information and sparse training data, it is usually difficult to achieve interpretability and high accuracy simultaneously for most methods. To address this difficulty, a new model is developed in this paper, namely Interpretable Multimodal Knowledge Graph Answer Prediction via Sequence Modeling (IMKGA-SM). First, a multi-modal fine-grained fusion method is proposed, and Vgg16 and Optical Character Recognition (OCR) techniques are adopted to effectively extract text information from images and images. Then, the knowledge graph link prediction task is modelled as an offline reinforcement learning Markov decision model, which is then abstracted into a unified sequence framework. An interactive perception-based reward expectation mechanism and a special causal masking mechanism are designed, which "converts" the query into an inference path. Then, an autoregressive dynamic gradient adjustment mechanism is proposed to alleviate the insufficient problem of multimodal optimization. Finally, two datasets are adopted for experiments, and the popular SOTA baselines are used for comparison. The results show that the developed IMKGA-SM achieves much better performance than SOTA baselines on multimodal link prediction datasets of different sizes.
NSF-MAP: Neurosymbolic Multimodal Fusion for Robust and Interpretable Anomaly Prediction in Assembly Pipelines
Artificial Intelligence Institute, University of South Carolina(南卡罗来纳大学人工智能研究所);Clemson Composites Center, Clemson University(克莱姆森大学复合材料中心);Intelligent Circuits, Architectures and Systems Lab, University of South Carolina(南卡罗来纳大学智能电路、架构和系统实验室)
Interpretable graph-based models on multimodal biomedical data integration: A technical review and benchmarking
Clemson University(克莱姆森大学);University of New England(新英格兰大学);UNSW Sydney(新南威尔士大学悉尼分校);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Tyree Foundation Institute of Health Engineering (IHealthE), UNSW Sydney(新南威尔士大学悉尼分校泰里基金会健康工程研究所)
Aetheria: A multimodal interpretable content safety framework based on multi-agent debate and collaboration
Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究院(TeleAI),中国电信);Sichuan University(四川大学);Peking University(北京大学);Beijing Jiaotong University(北京交通大学);Harbin Institute of Technology(哈尔滨工业大学);China Railway Rolling Stock Corporation Limited(中国铁路滚动股票有限公司)
Dynamic Summary Generation for Interpretable Multimodal Depression Detection
College of Information Science and Engineering, Ritsumeikan University(立命馆大学信息科学与工程学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Faculty of Engineering, University of the Ryukyus(琉球大学工学部)
Forecasting Clicks in Digital Advertising: Multimodal Inputs and Interpretable Outputs
Sony Group Corporation(索尼集团公司);Institute for Clarity in Documentation(清晰文档研究所);Inria Paris-Rocquencourt(巴黎- Rocquencourt 国家信息与自动化研究所);Rajiv Gandhi University(拉贾·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕勒尔研究实验室)
EAGLE: Efficient Alignment of Generalized Latent Embeddings for Multimodal Survival Prediction with Interpretable Attribution Analysis
Dept. of Machine Learning Moffitt Cancer Center(机器学习系莫菲特癌症中心);Dept. of Cancer Epidemiology Moffitt Cancer Center(癌症流行病学系莫菲特癌症中心);Dept. of Electrical Engineering University of South Florida(电气工程系佛罗里达州立大学)
Toward Interpretable Multimodal Fusion: Heat Conduction Modeling for Hyperspectral and LiDAR Joint Classification
Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院);Key Laboratory of Target Cognition and Application Technology(目标认知与应用技术重点实验室);School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences(中国科学院大学电子电气与通信工程学院);State Key Laboratory of Remote Sensing and Digital Earth, Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院遥感与数字地球国家重点实验室);Helmholtz-Zentrum Dresden-Rossendorf(亥姆霍兹德累斯顿罗森多夫研究中心);Faculty of Electrical and Computer Engineering, University of Iceland(冰岛大学电气与计算机工程系)
YOLO with Kolmogorov-Arnold networks and vision-language foundation models for interpretable object detection with trustworthy multimodal AI in computer vision perception
University of Bath(巴斯大学);Zhejiang University(浙江大学)
A Concept-based Interpretable Model for the Diagnosis of Choroid Neoplasias using Multimodal Data
University of Pennsylvania(宾夕法尼亚大学);University of Electronic Science and Technology(电子科技大学);Beijing Tongren Eye Center(北京同仁眼科中心);Beijing Tongren Hospital(北京同仁医院);Capital Medical University(首都医科大学)
TAMI: Temporally Aligned, Missingness-Aware, and Interpretable Multimodal Fusion for Mental Health Assessment in Older Adults with Mild Cognitive Impairment
Georgia Institute of Technology(佐治亚理工学院);Emory University(埃默里大学);Cairo University(开罗大学)
Interpretable Multimodal Gesture Recognition for Drone and Mobile Robot Teleoperation via Log-Likelihood Ratio Fusion
Department of Artificial Intelligence, Korea University(人工智能系,韩国大学);Department of Computer Science, RPTU Kaiserslautern-Landau(计算机科学系,RPTU凯撒斯劳滕-兰道);Embedded Intelligence, German Research Center for Artificial Intelligence (DFKI)(嵌入式智能,德国人工智能研究中心(DFKI))
ConceptM$^3$oE: Concept-Guided Multimodal Mixture of Experts for Interpretable Computational Pathology
University of Texas at Austin(德克萨斯大学奥斯汀分校);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Dell Children’s Medical Center(德尔儿童医疗中心);The University of Texas MD Anderson Cancer Center(德克萨斯大学MD安德森癌症中心);University of Nevada, Reno(内华达大学里诺分校)
Pan-Cancer Integrative Histology-Genomic Analysis via Interpretable Multimodal Deep Learning
Brigham and Women’s Hospital(布莱根妇女医院);Harvard Medical School(哈佛医学院);Broad Institute of Harvard and MIT(哈佛-麻省理工学院博德研究所);Dana-Farber Cancer Institute(丹娜-法伯癌症研究所);Harvard University(哈佛大学)
Interpretable Parallel Recurrent Neural Networks with Convolutional Attentions for Multi-Modality Activity Modeling
School of Computer Science and Engineering, UNSW Sydney, Australia(澳大利亚新南威尔士大学计算机科学与工程学院);School of Information Technology, RMIT University, Australia(澳大利亚皇家墨尔本理工大学信息技术学院);School of Computer Science, Northwestern Polytechnical University, China(中国西北工业大学计算机学院);School of Software, Tsinghua University China(中国清华大学软件学院)
Enabling Vision and Cross-Modal Learning for Multimodal Stroke Recurrence Prediction: An Interpretable Two-Step Framework
UMIT TIROL – Private University for Health Sciences and Health Technology(UMIT TIROL – 健康科学与健康技术私立大学);VASCage – Centre on Clinical Stroke Research(VASCage – 临床卒中研究中心);Medical University of Innsbruck(因斯布鲁克医科大学)
Interpretable Multimodal Learning for Cardiovascular Hemodynamics Assessment
University of Sheffield(谢菲尔德大学);Centre for Machine Intelligence (CMI)(机器智能中心);Department of Computer Science, University of Sheffield(谢菲尔德大学计算机科学系);Institute for in Silico Medicine (INSIGNEO)(计算机模拟医学研究所)
Platform of Bioinformatics (PBI), University Lausanne Hospital (CHUV) and Faculty of Biology and Medicine (FBM), University of Lausanne (UNIL)(生物信息平台(PBI)、日内瓦大学医院(CHUV)和生物医学学院(FBM)、日内瓦大学(UNIL));PBI, CHUV and FBM, UNIL(PBI、CHUV和FBM、UNIL);Department of Biomedical Engineering, University of Basel(生物医学工程系、巴塞尔大学);Roche Diagnostics International Ltd.(罗氏诊断国际有限公司);F. Hoffmann - La Roche Ltd., Roche Information Solutions(F. Hoffmann-La Roche有限公司、罗氏信息解决方案)
Concept-Grounded Reasoning with Prompt-Driven Localization for Interpretable Structured Report Generation
The Hong Kong University of Science and Technology(香港科技大学);The Chinese University of Hong Kong(香港中文大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Saarland University(萨尔兰大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of Cambridge(剑桥大学)
An interpretable generative multimodal neuroimaging-genomics framework for decoding Alzheimer's disease
Department of Computer Science, University of Verona(威尼斯大学计算机科学系);Department of Engineering for Innovation Medicine, University of Verona(威尼斯大学创新医学工程系);Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia State University, Georgia Institute of Technology, Emory University(神经影像与数据科学转化研究三机构中心(TReNDS),佐治亚州立大学,佐治亚理工学院,埃默里大学)
Center for Computing Research, Sandia National Laboratories(桑迪亚国家实验室计算研究中心);Applied Information Sciences Center, Sandia National Laboratories(桑迪亚国家实验室应用信息科学中心);School of Computing and Augmented Intelligence, Arizona State University(亚利桑那州立大学计算与增强智能学院);School of Engineering and Applied Science, University of Pennsylvania(宾夕法尼亚大学工程与应用科学学院)