arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.19752cs.CV
收藏

视觉事实核查器:实现高保真详细字幕生成

Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generation

NVIDIA(英伟达)

Yunhao Ge, Xiaohui Zeng, Jacob Samuel Huffman, Tsung-Yi Lin, Ming-Yu Liu, Yin Cui

AI总结:

本文提出无训练流程 VisualFactChecker,通过提议-验证-字幕生成三阶段结合开源模型与工具进行事实核查,在 2D 图像和 3D 物体字幕生成上以远小于专有模型的规模达到与 GPT-4V 相当的性能。

摘要:

现有的视觉内容自动字幕生成方法面临细节不足、内容幻觉和指令遵循能力差等挑战。本文提出 VisualFactChecker(VFC),一个灵活的无训练流程,可为 2D 图像和 3D 物体生成高保真且详细的字幕。VFC 包含三个步骤:1)提议阶段,图像到文本字幕模型提出多个初始字幕;2)验证阶段,大语言模型(LLM)利用目标检测和 VQA 模型等工具对提议字幕进行事实核查;3)字幕生成阶段,LLM 通过总结字幕提议和事实核查验证结果来生成最终字幕。在该阶段,VFC 可灵活地按照复杂指令生成多种风格的字幕。我们使用四项指标进行全面的字幕评估:1)CLIP-Score 用于衡量图像-文本相似度;2)CLIP-Image-Score 用于衡量原始图像与文本到图像模型根据字幕生成的重建图像之间的图像-图像相似度;3)在 Amazon Mechanical Turk 上进行人工研究;4)GPT-4V 用于细粒度评估。评估结果表明,在 COCO 数据集上的 2D 图像和 Objaverse 数据集上的 3D 资产方面,VFC 均优于最先进的开源字幕生成方法。我们的研究表明,通过将开源模型组合成一个流程,可以获得与 GPT-4V 等专有模型相当的字幕生成能力,尽管模型规模小 10 倍以上。

英文摘要:

Existing automatic captioning methods for visual content face challenges such as lack of detail, content hallucination, and poor instruction following. In this work, we propose VisualFactChecker (VFC), a flexible training-free pipeline that generates high-fidelity and detailed captions for both 2D images and 3D objects. VFC consists of three steps: 1) proposal, where image-to-text captioning models propose multiple initial captions; 2) verification, where a large language model (LLM) utilizes tools such as object detection and VQA models to fact-check proposed captions; 3) captioning, where an LLM generates the final caption by summarizing caption proposals and the fact check verification results. In this step, VFC can flexibly generate captions in various styles following complex instructions. We conduct comprehensive captioning evaluations using four metrics: 1) CLIP-Score for image-text similarity; 2) CLIP-Image-Score for measuring the image-image similarity between the original and the reconstructed image generated by a text-to-image model using the caption. 3) human study on Amazon Mechanical Turk; 4) GPT-4V for fine-grained evaluation. Evaluation results show that VFC outperforms state-of-the-art open-sourced captioning methods for 2D images on the COCO dataset and 3D assets on the Objaverse dataset. Our study demonstrates that by combining open-source models into a pipeline, we can attain captioning capability comparable to proprietary models such as GPT-4V, despite being over 10x smaller in model size.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑