面向零样本图像描述生成的基于视觉实体的可迁移解码
Transferable Decoding with Visual Entities for Zero-Shot Image Captioning
AI总结：
提出ViECap可迁移解码模型，通过实体感知硬提示引导LLMs聚焦图像视觉实体，有效缓解对象幻觉，在零样本跨域及域内图像描述生成任务中均取得领先性能。
摘要：
图像到文本生成旨在使用自然语言描述图像。近期，基于预训练视觉语言模型（VLMs）和大语言模型（LLMs）的零样本图像描述生成取得了显著进展。然而，我们观察并经实证证明，这些方法易受LLMs引发的模态偏差影响，倾向于生成包含训练期间频繁出现但图像中实际不存在的对象（实体）的描述（即对象幻觉）。本文提出ViECap，一种可迁移解码模型，利用实体感知解码在已见和未见场景中生成描述。ViECap结合实体感知硬提示，引导LLMs的注意力聚焦于图像中存在的视觉实体，实现跨多场景的连贯描述生成。借助实体感知硬提示，ViECap在从域内向域外场景迁移时能保持性能。大量实验表明，ViECap刷新了跨域（可迁移）描述生成的最优水平，且与以往基于VLMs的零样本方法相比，在域内描述生成中表现优异。代码已开源：https://github.com/FeiElysia/ViECap
英文摘要：
Image-to-text generation aims to describe images using natural language. Recently, zero-shot image captioning based on pre-trained vision-language models (VLMs) and large language models (LLMs) has made significant progress. However, we have observed and empirically demonstrated that these methods are susceptible to modality bias induced by LLMs and tend to generate descriptions containing objects (entities) that do not actually exist in the image but frequently appear during training (i.e., object hallucination). In this paper, we propose ViECap, a transferable decoding model that leverages entity-aware decoding to generate descriptions in both seen and unseen scenarios. ViECap incorporates entity-aware hard prompts to guide LLMs' attention toward the visual entities present in the image, enabling coherent caption generation across diverse scenes. With entity-aware hard prompts, ViECap is capable of maintaining performance when transferring from in-domain to out-of-domain scenarios. Extensive experiments demonstrate that ViECap sets a new state-of-the-art cross-domain (transferable) captioning and performs competitively in-domain captioning compared to previous VLMs-based zero-shot methods. Our code is available at: https://github.com/FeiElysia/ViECap
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
