GroundSight：利用定位信息与去幻觉增强视觉语言模型
GroundSight: Augmenting Vision-Language Models with Grounding Information and De-hallucination
AI总结：
本文提出 GroundSight，通过问题相关目标的边界框定位、裁剪检索及按问题类型去幻觉，提升 VQA 准确率并显著降低幻觉率。
摘要：
我们提出一种通过引入文本 grounded 的目标定位来改进基于检索增强生成（RAG）的视觉问答（VQA）的方法。与基于整幅图像检索信息不同，我们的方法使模型能够围绕与问题最相关的对象生成边界框，从而进行有针对性的图像裁剪和聚焦检索。这减少了背景噪声，改善了视觉线索与文本线索之间的对齐，并有助于缓解幻觉。我们的 RAG 方法增强了具备上下文感知能力的 VQA 回答；与基线 Llama-3.2-Vision-11B 智能体相比，准确率从 22.19% 提升至 25.64%，绝对提高 3.45 个百分点。我们还提出了一种基于问题类型的去幻觉方法，可有效将幻觉率从 65.79% 降至 13.88%，并提高真实性评分。
英文摘要：
We propose a method to improve Visual Question Answering (VQA) with Retrieval-Augmented Generation (RAG) by introducing text-grounded object localization. Rather than retrieving information based on the entire image, our approach enables the model to generate a bounding box around the object most relevant to the question, allowing for targeted image cropping and focused retrieval. This reduces background noise, improves alignment between visual and textual cues, and helps mitigate hallucinations. Our RAG method enhances context-aware VQA responses increased the accuracy from 22.19% to 25.64%, with an absolute increase of 3.45 percentage points, compared to the baseline Llama-3.2-Vision-11B agent. We also proposed a de-hallucination method based on question type which can effectively reduce the hallucination rate from 65.79% to 13.88% and improves the truthfulness score.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
