arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.09717cs.CV
收藏

视觉去幻觉指令生成:知道你不知道的

Visually Dehallucinative Instruction Generation: Know What You Don't Know

NCSOFT Corporation(NCSoft公司);NC Research(NC研究部)

Sungguk Cha, Jusung Lee, Younghyun Lee, Cheoljong Yang

AI总结:

本文提出视觉IK幻觉概念,构建VQAv2-IDK基准和IDK-Instructions数据库,并设计视觉去幻觉指令生成方法,有效减少多框架与数据集中的幻觉问题。

摘要:

“拿破仑皇帝何时发明了iPhone?”这类诱导幻觉的问题是生成式语言建模中众所周知的挑战。在本研究中,我们提出了视觉幻觉的创新概念,称为“I Know (IK)”幻觉,以应对期望回答“I Don't Know”的场景。为有效解决此问题,我们提出了VQAv2-IDK基准,它是VQAv2的子集,包含由人类标注者确定的不可回答的图像-问题对。进一步地,我们提出了针对IK幻觉的视觉去幻觉指令生成方法,并引入了IDK-Instructions视觉指令数据库。实验表明,当前方法难以应对IK幻觉。然而,我们的方法有效减少了这些幻觉,证明了其在不同框架和数据集上的通用性。

英文摘要:

"When did the emperor Napoleon invented iPhone?" Such hallucination-inducing question is well known challenge in generative language modeling. In this study, we present an innovative concept of visual hallucination, referred to as "I Know (IK)" hallucination, to address scenarios where "I Don't Know" is the desired response. To effectively tackle this issue, we propose the VQAv2-IDK benchmark, the subset of VQAv2 comprising unanswerable image-question pairs as determined by human annotators. Stepping further, we present the visually dehallucinative instruction generation method for IK hallucination and introduce the IDK-Instructions visual instruction database. Our experiments show that current methods struggle with IK hallucination. Yet, our approach effectively reduces these hallucinations, proving its versatility across different frameworks and datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑