arXiv 2502.110242025-02-16cs.CV

TPCap：利用触发增强和多模态净化模块解锁零样本图像描述

TPCap: Unlocking Zero-Shot Image Captioning with Trigger-Augmented and Multi-Modal Purification Modules

Faculty of Information Engineering and Automation, Kunming University of Science and Technology（昆明理工大学信息工程与自动化学院）；Yunnan Key Laboratory of Computer Technologies Application（云南省计算机技术应用重点实验室）；Hongyun Honghe Group Honghe Cigarette Factory（红云红河集团红河卷烟厂）

Ruoyu Zhang, Lulu Wang, Yi He, Tongling Pan, Zhengtao Yu, Yingna Li

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有基于RAG的图像描述方法依赖独立检索库导致计算开销大、限制LLMs零样本能力的问题，提出TPCap框架，通过触发增强生成和多模态净化模块，无需外部检索库实现零样本图像描述，在多数据集上取得与SOTA相当的性能。

摘要：

大型语言模型（LLMs）的最新进展显著提升了图像描述的流畅性和逻辑连贯性。检索增强生成（RAG）被广泛用于将外部知识融入LLMs；然而，现有基于RAG的方法依赖独立检索库，引入计算开销并限制LLMs固有零样本能力的利用。为解决这些局限，我们提出TPCap，一种无需外部检索库的新型触发增强和多模态净化框架，用于零样本图像描述。TPCap包含两个关键组件：触发增强（TA）生成和多模态净化（MP）。TA模块采用带有冻结和可学习投影的触发投射器，激活LLMs的上下文推理，增强视觉-文本对齐并减轻数据偏差。MP模块通过过滤噪声和提升特征质量进一步优化生成的实体相关信息，确保描述更精准且事实一致。我们在COCO、NoCaps、Flickr30k和WHOOPS数据集上评估TPCap。仅使用0.82M可训练参数并在单块NVIDIA RTX 4090 GPU上训练，TPCap实现了与最先进模型相当的竞争性能。

英文摘要：

Recent advancements in large language models (LLMs) have significantly enhanced the fluency and logical coherence of image captioning. Retrieval-Augmented Generation (RAG) is widely adopted to incorporate external knowledge into LLMs; however, existing RAG-based methods rely on separate retrieval banks, introducing computational overhead and limiting the utilization of LLMs' inherent zero-shot capabilities. To address these limitations, we propose TPCap, a novel trigger-augmented and multi-modal purification framework for zero-shot image captioning without external retrieval libraries. TPCap consists of two key components: trigger-augmented (TA) generation and multi-modal purification (MP). The TA module employs a trigger projector with frozen and learnable projections to activate LLMs' contextual reasoning, enhance visual-textual alignment, and mitigate data bias. The MP module further refines the generated entity-related information by filtering noise and enhancing feature quality, ensuring more precise and factually consistent captions. We evaluate TPCap on COCO, NoCaps, Flickr30k, and WHOOPS datasets. With only 0.82M trainable parameters and training on a single NVIDIA RTX 4090 GPU, TPCap achieves competitive performance comparable to state-of-the-art models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。