FaithScore：大型视觉语言模型中幻觉的细粒度评估
FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Models
AI总结：
本文提出FaithScore，一种无需参考的细粒度评估指标，通过提取原子事实并与图像进行一致性验证，衡量大型视觉语言模型生成答案的忠实度，揭示当前模型存在幻觉问题。
摘要：
我们引入了FaithScore（Faithfulness to Atomic Image Facts Score，原子图像事实忠实度评分），这是一种无需参考的细粒度评估指标，用于衡量大型视觉语言模型（LVLMs）生成的自由形式答案的忠实度。FaithScore评估首先识别包含需要验证的描述性陈述的子句，然后从这些子句中提取详尽的原子事实列表，最后在细粒度原子事实与输入图像之间进行一致性验证。元评估表明，我们的指标与人类对忠实度的判断高度相关。我们收集了两个基准数据集（即LLaVA-1k和MSCOCO-Cap），用于评估LVLMs的指令遵循幻觉。我们使用FaithScore在这些数据集上测量了最先进的LVLMs中的幻觉。结果显示，当前系统容易生成与图像不符的幻觉内容，这为未来的改进留下了空间。我们希望我们的指标FaithScore能帮助评估未来LVLMs的忠实度，并为增强LVLMs的忠实度提供有见地的建议。
英文摘要：
We introduce FaithScore (Faithfulness to Atomic Image Facts Score), a reference-free and fine-grained evaluation metric that measures the faithfulness of the generated free-form answers from large vision-language models (LVLMs). The FaithScore evaluation first identifies sub-sentences containing descriptive statements that need to be verified, then extracts a comprehensive list of atomic facts from these sub-sentences, and finally conducts consistency verification between fine-grained atomic facts and the input image. Meta-evaluation demonstrates that our metric highly correlates with human judgments of faithfulness. We collect two benchmark datasets (i.e. LLaVA-1k and MSCOCO-Cap) for evaluating LVLMs instruction-following hallucinations. We measure hallucinations in state-of-the-art LVLMs with FaithScore on the datasets. Results reveal that current systems are prone to generate hallucinated content unfaithful to the image, which leaves room for future improvements. We hope our metric FaithScore can help evaluate future LVLMs in terms of faithfulness and provide insightful advice for enhancing LVLMs' faithfulness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
