使VLM在卡通角色图像上通过姿态信息识别视觉幻觉
Make VLM Recognize Visual Hallucination on Cartoon Character Image with Pose Information
AI总结：
本文提出基于姿态信息的VLM幻觉检测方法，通过上下文学习提升识别准确率，实验表明在卡通角色图像中幻觉检测效果提升50%-80%。
摘要：
利用大规模文本到图像（TTI）模型已成为在图像合成、视频编辑和3D重建领域生成示例或训练数据的常见技术。然而，涉及感知严重缺陷的语义结构视觉幻觉仍是一个问题，尤其是在非逼真渲染（NPR）领域，如卡通和像素化风格角色。为了检测这些幻觉，我们提出了一种使用视觉语言模型（VLM）的新型语义结构幻觉检测系统。我们的方法是利用大语言模型的新兴能力，即上下文学习，即VLM通过用户为特定下游任务看到了一些示例，这里就是幻觉检测。基于上下文学习，我们引入了姿态感知的上下文视觉学习（PA-ICVL），通过进一步输入视觉数据，除了提示、RGB图像和姿态信息，从而提高VLM的整体性能。通过结合姿态指导，我们使VLM能够做出更准确的决策。实验结果表明，与仅依赖RGB图像的基线方法相比，在识别视觉幻觉方面有显著改进。在选定的两个VLM中，GPT-4v、Gemini Pro Vision，我们提出的PA-ICVL在幻觉检测方面分别提高了50%到78%、57%到80%。这项研究通过上下文视觉学习减轻视觉幻觉，推动TTI模型向实际应用迈进，扩展了其在非逼真领域的潜力。此外，它展示了用户如何通过利用额外条件来增强开放VLM的下游专业化能力。我们收集了合成的卡通幻觉数据集，该数据集和最终调优的VLM将公开可用。
英文摘要：
Leveraging large-scale Text-to-Image (TTI) models have become a common technique for generating exemplar or training dataset in the fields of image synthesis, video editing, 3D reconstruction. However, semantic structural visual hallucinations involving perceptually severe defects remain a concern, especially in the domain of non-photorealistic rendering (NPR) such as cartoons and pixelization-style character. To detect these hallucinations in NPR, We propose a novel semantic structural hallucination detection system using Vision-Language Model (VLM). Our approach is to leverage the emerging capability of large language model, in-context learning which denotes that VLM has seen some examples by user for specific downstream task, here hallucination detection. Based on in-context learning, we introduce pose-aware in-context visual learning (PA-ICVL) which improve the overall performance of VLM by further inputting visual data beyond prompts, RGB images and pose information. By incorporating pose guidance, we enable VLMs to make more accurate decisions. Experimental results demonstrate significant improvements in identifying visual hallucinations compared to baseline methods relying solely on RGB images. Within selected two VLMs, GPT-4v, Gemini pro vision, our proposed PA-ICVL improves the hallucination detection with 50% to 78%, 57% to 80%, respectively. This research advances a capability of TTI models toward real-world applications by mitigating visual hallucinations via in-context visual learning, expanding their potential in non-photorealistic domains. In addition, it showcase how users can boost the downstream-specialized capability of open VLM by harnessing additional conditions. We collect synthetic cartoon-hallucination dataset with TTI models, this dataset and final tuned VLM will be publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
