通过使用改写字幕微调大型视觉语言模型来缓解细粒度幻觉
Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites
AI总结：
本文提出 ReCaption 框架，通过 ChatGPT 改写字幕并微调 LVLMs，同时设计 FGHE 细粒度评估方法，有效减少细粒度对象幻觉并提升文本生成质量。
摘要：
大型语言模型（LLMs）在自然语言处理（NLP）任务中表现出卓越的性能。为了理解并执行针对图像数据的多样化人类指令，指令微调的大型视觉语言模型（LVLMs）被提出。然而，LVLMs 可能会遭受不同类型的对象幻觉。尽管如此，目前对 LVLMs 的评估仅针对粗粒度对象幻觉（即生成的物体在输入图像中不存在）。图像中不存在的细粒度对象属性和行为仍可能被生成，但当前的评估方法无法测量这些内容。因此，本文聚焦于减少 LVLMs 的细粒度幻觉。我们提出了 ReCaption 框架，该框架由两个组件组成：使用 ChatGPT 改写字幕，并在改写后的字幕上微调指令微调的 LVLMs。我们还提出了一种基于细粒度探测的评估方法，名为细粒度对象幻觉评估（FGHE）。我们的实验结果表明，ReCaption 能有效减少不同 LVLM 选项的细粒度对象幻觉，并提升其文本生成质量。代码可在 https://github.com/Anonymousanoy/FOHE 找到。
英文摘要：
Large language models (LLMs) have shown remarkable performance in natural language processing (NLP) tasks. To comprehend and execute diverse human instructions over image data, instruction-tuned large vision-language models (LVLMs) have been introduced. However, LVLMs may suffer from different types of object hallucinations. Nevertheless, LVLMs are evaluated for coarse-grained object hallucinations only (i.e., generated objects non-existent in the input image). The fine-grained object attributes and behaviors non-existent in the image may still be generated but not measured by the current evaluation methods. In this paper, we thus focus on reducing fine-grained hallucinations of LVLMs. We propose \textit{ReCaption}, a framework that consists of two components: rewriting captions using ChatGPT and fine-tuning the instruction-tuned LVLMs on the rewritten captions. We also propose a fine-grained probing-based evaluation method named \textit{Fine-Grained Object Hallucination Evaluation} (\textit{FGHE}). Our experiment results demonstrate that ReCaption effectively reduces fine-grained object hallucination for different LVLM options and improves their text generation quality. The code can be found at https://github.com/Anonymousanoy/FOHE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
