视觉去幻觉指令生成
Visually Dehallucinative Instruction Generation
AI总结：
本文提出CAP2QA方法，通过将生成范围严格约束为仅限图像内容，有效减少生成式语言模型在视觉指令中的幻觉问题，并构建了图像对齐的指令式问答数据集CAP2QA-COCO，实验证明该方法在显著降低视觉幻觉的同时提升了视觉识别能力与表达能力。
摘要：
近年来，生成式语言模型生成的合成视觉指令在视觉问答任务上表现出合理的文本生成性能。然而，生成式语言模型的幻觉问题依然存在，即生成的图文数据包含非预期内容。本文提出了一种新颖且可扩展的视觉去幻觉指令生成方法，命名为CAP2QA，该方法将生成范围严格约束为仅限图像内容。我们的核心贡献在于引入了图像对齐的指令式问答数据集CAP2QA-COCO及其可扩展的构建方案。在实验中，我们通过视觉指令微调对基于相同源数据的合成视觉指令数据集进行了比较，并执行了通用视觉识别任务。结果表明，我们提出的方法显著减少了视觉幻觉，同时持续提升了视觉识别能力与表达能力。
英文摘要：
In recent years, synthetic visual instructions by generative language model have demonstrated plausible text generation performance on the visual question-answering tasks. However, challenges persist in the hallucination of generative language models, i.e., the generated image-text data contains unintended contents. This paper presents a novel and scalable method for generating visually dehallucinative instructions, dubbed CAP2QA, that constrains the scope to only image contents. Our key contributions lie in introducing image-aligned instructive QA dataset CAP2QA-COCO and its scalable recipe. In our experiments, we compare synthetic visual instruction datasets that share the same source data by visual instruction tuning and conduct general visual recognition tasks. It shows that our proposed method significantly reduces visual hallucination while consistently improving visual recognition ability and expressiveness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
