NLKI：一个用于提升小型 VLM 在常识 VQA 任务中表现的轻量级自然语言知识整合框架
NLKI: A lightweight Natural Language Knowledge Integration Framework for Improving Small VLMs in Commonsense VQA Tasks
AI总结：
提出 NLKI 轻量框架，通过 ColBERTv2 事实检索、LLM 解释生成与噪声鲁棒微调增强小型 VLM，在常识 VQA 等任务上最高提升 7% 准确率。
摘要：
常识视觉问答往往依赖图像或问题中缺失的知识。因此，ViLT、VisualBERT 和 FLAVA 等小型视觉语言模型（sVLM）落后于规模更大的生成式同类模型。为研究审慎的常识知识整合对 sVLM 的影响，我们提出端到端框架 NLKI：（i）检索自然语言事实，（ii）提示 LLM 生成自然语言解释，（iii）将两类信号分别输入 sVLM，并在两个常识 VQA 数据集（CRIC、AOKVQA）和一个视觉蕴含数据集（e-SNLI-VE）上评估。使用经过微调的 ColBERTv2 检索事实，并采用富含对象信息的提示，所得解释大幅减少幻觉，同时在 3 个数据集上将端到端答案准确率最高提升 7%，使 NLKI 中的 FLAVA 及其他模型达到或超过 Qwen-2 VL-2B 和 SmolVLM-2.5B 等中型 VLM。由于这些基准包含 10%–25% 的标签噪声，使用对称交叉熵和广义交叉熵等噪声鲁棒损失进行额外微调，又在 CRIC 上带来 2.5%、在 AOKVQA 上带来 5.5% 的提升。我们的发现揭示了基于 LLM 的常识知识何时优于从常识知识库检索、噪声感知训练如何在外部知识增强情境下稳定小型模型，以及为什么 250M 模型如今已能实现参数高效的常识推理。
英文摘要：
Commonsense visual-question answering often hinges on knowledge that is missing from the image or the question. Small vision-language models (sVLMs) such as ViLT, VisualBERT and FLAVA therefore lag behind their larger generative counterparts. To study the effect of careful commonsense knowledge integration on sVLMs, we present an end-to-end framework (NLKI) that (i) retrieves natural language facts, (ii) prompts an LLM to craft natural language explanations, and (iii) feeds both signals to sVLMs respectively across two commonsense VQA datasets (CRIC, AOKVQA) and a visual-entailment dataset (e-SNLI-VE). Facts retrieved using a fine-tuned ColBERTv2 and an object information-enriched prompt yield explanations that largely cut down hallucinations, while lifting the end-to-end answer accuracy by up to 7% (across 3 datasets), making FLAVA and other models in NLKI match or exceed medium-sized VLMs such as Qwen-2 VL-2B and SmolVLM-2.5B. As these benchmarks contain 10-25% label noise, additional finetuning using noise-robust losses (such as symmetric cross entropy and generalised cross entropy) adds another 2.5% in CRIC, and 5.5% in AOKVQA. Our findings expose when LLM-based commonsense knowledge beats retrieval from commonsense knowledge bases, how noise-aware training stabilises small models in the context of external knowledge augmentation, and why parameter-efficient commonsense reasoning is now within reach for 250M models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
