Multi-modal knowledge graphs (MKGs) include not only the relation triplets, but also related multi-modal auxiliary data (i.e., texts and images), which enhance the diversity of knowledge. However, the natural incompleteness has significantly hindered the applications of MKGs. To tackle the problem, existing studies employ the embedding-based reasoning models to infer the missing knowledge after fusing the multi-modal features. However, the reasoning performance of these methods is limited due to the following problems: (1) ineffective fusion of multi-modal auxiliary features; (2) lack of complex reasoning ability as well as inability to conduct the multi-hop reasoning which is able to infer more missing knowledge. To overcome these problems, we propose a novel model entitled MMKGR (Multi-hop Multi-modal Knowledge Graph Reasoning). Specifically, the model contains the following two components: (1) a unified gate-attention network which is designed to generate effective multi-modal complementary features through sufficient attention interaction and noise reduction; (2) a complementary feature-aware reinforcement learning method which is proposed to predict missing elements by performing the multi-hop reasoning process, based on the features obtained in component (1). The experimental results demonstrate that MMKGR outperforms the state-of-the-art approaches in the MKG reasoning task.
Overview of the NLPCC 2025 Shared Task 4: Multi-modal, Multilingual, and Multi-hop Medical Instructional Video Question Answering Challenge
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究所，中国科学院）；School of Computer Science and Technology, Beijing Institute of Technology（计算机科学与技术学院，北京理工大学）；School of Engineering, Westlake University（工程学院，西湖大学）
M$^3$-Med: A Benchmark for Multi-lingual, Multi-modal, and Multi-hop Reasoning in Medical Instructional Video Understanding
Beijing Institute of Technology（北京理工大学）；The Hong Kong Polytechnic University（香港理工大学）；Chinese Academy of Sciences（中国科学院）；Air Force Hospital of Southern Theater Command of PLA（中国人民解放军南部战区空军医院）；Shenzhen Traditional Chinese Medicine Hospital（深圳中医药医院）
MultiModalQA: Complex Question Answering over Text, Tables and Images
The Allen Institute for AI（艾伦人工智能研究所）；Tel-Aviv University（特拉维夫大学）；University of Washington（华盛顿大学）
在回答复杂的问题时，人们可以无缝地结合来自视觉，文本和表格来源的信息。虽然近年来对多个证据推理的模型的兴趣激增，但对跨多个模态推理的问答模型的研究相对较少。在本文中，我们提出了MultiModalQA（MMQA）：一个具有挑战性的问答数据集，需要联合推理的文本，表格和图像。我们使用一个新的框架来创建MMQA，该框架用于大规模生成复杂的多模态问题，从维基百科获取表格，并使用每个表格中出现的实体附加图像和文本段落。然后，我们定义了一个形式语言，使我们能够采取的问题，可以回答从一个单一的模态，并将它们结合起来，以产生跨模态的问题。最后，众包工作人员将这些自动生成的问题转化为更流畅的语言。我们通过这个过程创建了29，918个问题，并以经验证明了多模态多跳方法解决我们任务的必要性：我们的多跳模型ImplicitDecomp在跨模态问题上的平均F1为51.7，大大超过了达到38.2 F1的强大基线，但仍然明显落后于人类表现，这是90.1 F1
Stanford University（斯坦福大学）；Intel AI Lab（英特尔人工智能实验室）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；Mila - Québec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）
A Comprehensive Survey on Multi-modal Conversational Emotion Recognition with Deep Learning
Central South University of Forestry and Technology（中部林业科技大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；State University of New York（纽约州立大学）
Red-teaming the Multimodal Reasoning: Jailbreaking Vision-Language Models via Cross-modal Entanglement Attacks
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；People’s Public Security University of China（中国人民公安大学）
Explore the Limits of Omni-modal Pretraining at Scale
The Chinese University of Hong Kong（香港中文大学）；University of Chinese Academy of Sciences（中国科学院大学）；Chinese Academy of Science（中国科学院）；Shanghai AI Laboratory（上海人工智能实验室）
CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models
School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；Key Laboratory of Data Intelligence and Advanced Computing in Provincial Universities, Soochow University（苏州大学省级高校数据智能与先进计算重点实验室）；Research Center for SCIR, Harbin Institute of Technology（哈尔滨工业大学SCIR研究中心）；National University of Singapore（新加坡国立大学）
Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Framework
Westlake University（西湖大学）；Shenyang Institute of Computing Technology, Chinese Academy of Sciences（中国科学院沈阳计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Natural language is not enough: Benchmarking multi-modal generative AI for Verilog generation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（中国科学院大学杭州高等研究院）；Shanghai Jiao Tong University（上海交通大学）
METER: Multi-modal Evidence-based Thinking and Explainable Reasoning -- Algorithm and Benchmark
Institute of Artificial Intelligence (TeleAI), China Telecom（电信人工智能研究院）；Institute of Artificial Intelligence and Robotics(IAIR), Xi’an Jiaotong University（人工智能与机器人研究院）；Advanced Technique of Artificial Intelligence(ATAI), Chongqing University of Technology（人工智能先进技术研究院）
M$^3$CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought
Research Center for Social Computing and Information Retrieval（社会计算与信息检索研究中心）；Harbin Institute of Technology（哈尔滨工业大学）；School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；Shanghai AI Laboratory（上海人工智能实验室）
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Technology Sydney（悉尼科技大学）；Northeastern University（东北大学）；University of Liverpool（利物浦大学）
MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education
AI Singapore, National University of Singapore（新加坡国立大学新加坡人工智能）；School of Computing, National University of Singapore（新加坡国立大学计算学院）；Institute of Advanced Intelligence and Computing, A*STAR（新加坡科技研究局先进智能与计算研究所）
