Multimodal knowledge graphs (MMKGs) organize information across different modalities to support multimodal Graph Retrieval-Augmented Generation (RAG), which aims to improve the performance of multimodal large language models (MLLMs). The effectiveness of these multimodal Graph RAG methods largely depends on the quality of the constructed MMKGs. However, most existing methods construct MMKGs without explicit guidance on graph quality, while reinforcement learning (RL)-based methods that rely on final-answer correctness may provide only indirect feedback on MMKG quality. To address these limitations, we propose MMKG-R1, a novel MMKG-quality-driven RL framework for training an MLLM as a multimodal knowledge constructor. To be specific, MMKG-R1 introduces multiple quality-aware rewards to directly evaluate both the quality of information extracted from individual knowledge sources and the quality of the assembled MMKG. These rewards are then used to guide the RL training. Experiments across multiple datasets show that MMKG-R1 constructs MMKGs that improve downstream question-answering accuracy.
Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning
Tongji University(同济大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);East China Normal University(华东师范大学);Stanford University(斯坦福大学);New York University(纽约大学)
本文提出 VaLiK 方法,通过跨模态信息补充构建无需注释的多模态知识图谱,提升大语言模型推理能力。
NKIARI & SLAI, Shenzhen Futian and VCIP, Nankai University, Tianjin, China(NKIARI与SLAI,深圳福田及VCIP,南开大学,天津,中国);Australian National University (ANU), Canberra, Australia(澳大利亚国立大学(ANU),堪培拉,澳大利亚);King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia(国王阿卜杜勒·阿齐兹大学(KAUST),图瓦尔,沙特阿拉伯);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore(高性能计算研究所(IHPC),科学、技术和研究局(A*STAR),新加坡)
Sparse Shortcuts: Facilitating Efficient Fusion in Multimodal Large Language Models
Guangdong-Hong Kong-Macao Joint Laboratory for Emotion Intelligence and Pervasive Computing, Artificial Intelligence Research Institute, Shenzhen MSU-BIT University, Shenzhen(粤港澳大湾区情感智能与 pervasive 计算联合实验室,人工智能研究院,深圳MSU-BIT大学)
Can Multimodal Large Language Models Truly Understand Small Objects?
Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学);MMLab, The Chinese University of Hong Kong(香港中文大学MMLab)
City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Zhejiang University(浙江大学);Fudan University(复旦大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Terminus Technologies Co., Ltd.;Pengcheng Laboratory(鹏城实验室)
Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models
School of Cyber Science and Engineering, Southeast University(东南大学网络空间科学与工程学院);College of Artificial Intelligence and Automation, Hohai University(河海大学人工智能与自动化学院);School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院)
Multimodal Graph RAG for Long-range Visually Rich Document Understanding
Department of Computer Science and Information Engineering, National Taiwan University(国家台湾大学计算机科学与信息工程系);FinTech Center, National Taiwan University(国家台湾大学金融科技中心)
MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research
Stanford University(斯坦福大学);Tsinghua University(清华大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Princeton University(普林斯顿大学);KTH Royal Institute of Technology(瑞典皇家理工学院);Chan Zuckerberg Biohub Network(陈·扎克翰生物中心网络)
BuildingView: Constructing Urban Building Exteriors Databases with Street View Imagery and Multimodal Large Language Mode
Thrust of Urban Governance and Design, The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)城市治理与设计中心);Spatial Sciences Institute, University of Southern California(南加州大学空间科学学院)
Enhancing Multimodal Medical Image Classification using Cross-Graph Modal Contrastive Learning
Department of Systems Engineering, Stevens Institute of Technology, Hoboken, New Jersey, USA(系统工程系,史蒂文斯理工学院);Department of Nuclear Medicine, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung, Taiwan(高雄长庚纪念医院核医学部);Institute of Statistics, National University of Kaohsiung, Kaohsiung, Taiwan(国立高雄大学统计研究所);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China(深圳先进技术研究院,中国科学院)