arXiv 2505.036542026-09-01cs.CVcs.AI

ReGraP-LLaVA：支持推理的基于图的个性化多模态大语言与视觉助手

ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant

Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；School of Engineering, Westlake University（西湖大学工程学院）；University of Minnesota – Twin Cities（明尼苏达大学双城分校）；University of Toronto（多伦多大学）

Yifan Xiang, Zhenxi Zhang, Bin Li, Yixuan Weng, Bo Gao, Shoujun Zhou, Yangfan He, Yilin Yuan, Keqin Li

备注：: accepted in EMNLP 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有个性化多模态大语言模型忽略概念关联的局限，本文构建ReGraP数据集与基准，提出ReGraP-LLaVA模型，实现个性化关系推理，在对比基准中性能最优。

摘要：

多模态大语言模型（MLLM）已在多模态任务中展现出优异性能，现有个性化MLLM可识别用户特定概念并生成上下文相关描述文本。但现有个性化MLLM主要聚焦孤立概念，常缺乏关系训练数据，忽略个性化概念间的关联，且多在识别或描述生成任务上评估。为解决这些局限，本文构建了ReGraP数据集，包含120个个性化知识集，每个知识集涵盖图像、知识图谱（KG）及思维链问答（CoT QA）对。基于ReGraP，本文提出ReGraP-LLaVA，这是一款通过软或硬图提示融入KG与CoT QA对的个性化MLLM，用于将结构化关系知识与模型语义空间对齐。本文还建立了ReGraP基准，涵盖开放与封闭场景下的选择题、填空题、判断题及描述题，以评估个性化关系推理与知识关联能力。实验结果表明，ReGraP-LLaVA可有效学习个性化知识并执行关系推理，在对比基准中取得了最佳整体性能。代码与数据可在指定网址获取。

英文摘要：

Multimodal Large Language Models have shown strong performance across multimodal tasks, and recent personalized MLLMs can recognize user-specific concepts and generate contextual captions. However, existing personalized MLLMs mainly focus on isolated concepts, often lacking relational training data, neglecting connections among personalized concepts, and evaluating mostly on recognition or captioning. To address these limitations, we introduce ReGraP, a dataset of 120 personalized knowledge sets, each containing images, knowledge graphs, and Chain-of-Thought Question-Answering pairs. Based on ReGraP, we propose Reasoning enabled Graph-based Personalized Large Language and Vision Assistant ReGraP-LLaVA, a personalized MLLM that incorporates KGs and CoT QA pairs through soft and/or hard graph prompting to align structured relational knowledge with the model's semantic space. We further establish the ReGraP Benchmark, covering multiple-choice, fill-in-the-blank, true/false, and descriptive questions in both open- and closed-ended settings, to evaluate personalized relational reasoning and knowledge-connection capabilities. Experimental results show that ReGraP-LLaVA effectively learns personalized knowledge and performs relational reasoning, achieving the best overall performance among competitive baselines. Code and data are available at: https://github.com/xyfyyds/ReGraP

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。