弥合视觉差距：使用知识适应的描述微调多模态模型
Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions
AI总结：
针对小规模视觉语言模型微调时图像描述丰富性与幻觉的平衡问题，提出DNLI评估框架并引入KnowAda微调方法，利用模型已有知识自动调整训练数据，在最高7B参数模型上有效降低了幻觉并保持了高描述性。
摘要：
近期研究越来越关注使用长篇详细的图像描述来训练视觉语言模型（VLMs）。然而，小规模 VLMs 在微调时往往难以平衡这些描述的丰富性与产生内容幻觉的风险。本文探讨了 VLMs 对此类描述的适应程度。为量化描述质量，我们提出了 Decomposed NLI (DNLI)，这是一个评估框架，将生成的描述分解为单个命题，并逐一进行孤立评估。这种细粒度分析揭示了捕捉描述细节与防止幻觉之间的关键平衡。我们的发现表明，仅仅降低描述复杂性或采用标准数据筛选技术并不能有效解决此问题。为应对这一挑战，我们引入了 Knowledge Adapted (KnowAda) 微调，这是一种以数据为中心的方法，利用模型已有知识和视觉理解自动调整训练数据。KnowAda 在保持高描述性的同时最小化了幻觉。我们在多个小规模 VLMs（参数量最高达 7B）和密集描述数据集上验证了该方法，证明 KnowAda 有效平衡了幻觉减少与描述性。结果表明，KnowAda 在自动指标和人类评估中均优于各种基线方法。我们将发布代码和模型。
英文摘要：
Recent research increasingly focuses on training vision-language models (VLMs) with long, detailed image captions. However, small-scale VLMs often struggle to balance the richness of these captions with the risk of hallucinating content during fine-tuning. In this paper, we explore how well VLMs adapt to such captions. To quantify caption quality, we propose Decomposed NLI (DNLI), an evaluation framework that breaks down generated captions into individual propositions, assessing each in isolation. This fine-grained analysis reveals a critical balance between capturing descriptive details and preventing hallucinations. Our findings show that simply reducing caption complexity or employing standard data curation techniques does not effectively resolve this issue. To tackle this challenge, we introduce Knowledge Adapted (KnowAda) fine-tuning, a data-centric approach that automatically adapts training data with the model's existing knowledge and visual understanding. KnowAda minimizes hallucinations while preserving high descriptiveness. We validate this approach across several small-scale VLMs (up to 7B parameters) and dense caption datasets, demonstrating that KnowAda effectively balances hallucination reduction and descriptiveness. Our results show that KnowAda outperforms various baselines in both automatic metrics and human evaluations. We will release our code and models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
