超越通用：使用视觉-语言预训练模型融入真实世界知识以增强图像描述
Beyond Generic: Enhancing Image Captioning with Real-World Knowledge using Vision-Language Pre-Training Model
AI总结：
针对图像描述缺乏真实世界知识的问题，提出 Knowledge-guided Replay 方法，通过知识预测和蒸馏约束在微调中保留预训练知识，有效缓解幻觉，在 KnowCap 基准上显著提升知识识别准确率与描述质量。
摘要：
当前的图像描述方法往往生成正确但“通用”的描述，缺乏真实世界知识，例如命名实体和上下文信息。考虑到视觉-语言预训练（VLP）模型从大规模网络抓取数据中掌握了海量此类知识，利用 VLP 模型的泛化能力将知识融入图像描述极具前景。然而，使用 VLP 模型面临挑战：零样本推理会因知识幻觉导致低质量描述，而下游任务微调中的通用偏见会阻碍 VLP 模型表达知识。为解决这些问题，我们提出一种简单有效的方法 Knowledge-guided Replay (K-Replay)，在微调期间保留预训练知识。该方法包含两部分：(1) 在自动收集的重放样本上进行知识预测任务，持续唤醒 VLP 模型对知识的记忆，防止模型退化为通用模式；(2) 知识蒸馏约束以提高生成描述的忠实度，从而缓解知识幻觉。为评估知识增强的描述，我们构建了新颖的图像描述基准 KnowCap，包含地标、著名品牌、特色食品和电影角色的知识。实验结果表明，该方法有效将知识融入描述，在 CIDEr 分数上较强大的 VLP 基线提升 20.9 分（78.7->99.6），在知识识别准确率上提升 20.5 个百分点（34.0%->54.5%）。
英文摘要：
Current captioning approaches tend to generate correct but "generic" descriptions that lack real-world knowledge, e.g., named entities and contextual information. Considering that Vision-Language Pre-Training (VLP) models master massive such knowledge from large-scale web-harvested data, it is promising to utilize the generalizability of VLP models to incorporate knowledge into image descriptions. However, using VLP models faces challenges: zero-shot inference suffers from knowledge hallucination that leads to low-quality descriptions, but the generic bias in downstream task fine-tuning hinders the VLP model from expressing knowledge. To address these concerns, we propose a simple yet effective method called Knowledge-guided Replay (K-Replay), which enables the retention of pre-training knowledge during fine-tuning. Our approach consists of two parts: (1) a knowledge prediction task on automatically collected replay exemplars to continuously awaken the VLP model's memory about knowledge, thus preventing the model from collapsing into the generic pattern; (2) a knowledge distillation constraint to improve the faithfulness of generated descriptions hence alleviating the knowledge hallucination. To evaluate knowledge-enhanced descriptions, we construct a novel captioning benchmark KnowCap, containing knowledge of landmarks, famous brands, special foods and movie characters. Experimental results show that our approach effectively incorporates knowledge into descriptions, outperforming strong VLP baseline by 20.9 points (78.7->99.6) in CIDEr score and 20.5 percentage points (34.0%->54.5%) in knowledge recognition accuracy. Our code and data is available at https://github.com/njucckevin/KnowCap.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
