基于指纹检索与推理的无训练个性化
Training-Free Personalization via Retrieval and Reasoning on Fingerprints
AI总结：
针对视觉语言模型难以理解用户特定概念的问题，提出无训练个性化方法R2P，通过提取概念指纹、检索与思维链推理及跨模态验证，在多个基准上优于现有方法。
摘要：
视觉语言模型（VLMs）在多模态推理方面带来了重大改进，但它们仍然难以理解用户特定的概念。现有的个性化方法解决了这一局限性，但严重依赖于训练过程，这些过程可能对个体用户而言成本高昂或令人不适。我们不同于现有工作，首次在个性化背景下探索无训练设置。我们提出了一种新方法，即用于个性化的检索与推理（R2P），利用VLMs的内部知识。首先，我们利用VLMs提取概念指纹，即在其语义类别内唯一定义该概念的关键属性。当查询到来时，检索最相似的指纹，并通过思维链推理进行评分。为降低幻觉风险，评分通过属性级别的跨模态验证进行校验：在评分不一致的情况下，R2P通过成对多模态匹配来细化概念关联，其中检索到的指纹及其图像与查询直接进行比较。我们在两个公开基准和一个新引入的数据集（具有视觉模糊性的个人概念，PerVA）上验证了R2P，该数据集用于突出视觉模糊性挑战的概念识别。在所有基准的各种下游任务中，R2P始终优于最先进的方法。代码将在接受后提供。
英文摘要：
Vision Language Models (VLMs) have lead to major improvements in multimodal reasoning, yet they still struggle to understand user-specific concepts. Existing personalization methods address this limitation but heavily rely on training procedures, that can be either costly or unpleasant to individual users. We depart from existing work, and for the first time explore the training-free setting in the context of personalization. We propose a novel method, Retrieval and Reasoning for Personalization (R2P), leveraging internal knowledge of VLMs. First, we leverage VLMs to extract the concept fingerprint, i.e., key attributes uniquely defining the concept within its semantic class. When a query arrives, the most similar fingerprints are retrieved and scored via chain-of-thought-reasoning. To reduce the risk of hallucinations, the scores are validated through cross-modal verification at the attribute level: in case of a discrepancy between the scores, R2P refines the concept association via pairwise multimodal matching, where the retrieved fingerprints and their images are directly compared with the query. We validate R2P on two publicly available benchmarks and a newly introduced dataset, Personal Concepts with Visual Ambiguity (PerVA), for concept identification highlighting challenges in visual ambiguity. R2P consistently outperforms state-of-the-art approaches on various downstream tasks across all benchmarks. Code will be available upon acceptance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
