arXiv 2307.055912025-02-08cs.CVcs.CLcs.LG

用于图像字幕生成的视觉-语言模型线性对齐

Linear Alignment of Vision-language Models for Image Captioning

ELLIS Unit Linz（ELLIS林茨单元）；LIT AI Lab（LIT人工智能实验室）；Institute for Machine Learning（机器学习研究所）；JKU LIT SAL（约翰内斯·开普勒大学林茨软件应用实验室）；Johannes Kepler University（约翰内斯·开普勒大学）

Fabian Paischer, Markus Hofmarcher, Sepp Hochreiter, Thomas Adler

备注：: 9 pages (+ references and appendix)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出ReCap方法，通过闭式线性映射重新对齐CLIP联合嵌入空间，实现训练速度提升1000倍的轻量级图像字幕生成，并提出两种新的CLIP评估指标，在多个基准上取得领先性能。

摘要：

近年来，像CLIP这样的视觉-语言模型在图像字幕生成和字幕评估等多种多模态任务中推动了最先进水平的发展。许多方法利用CLIP进行跨模态检索，以视觉输入为条件来调节预训练语言模型。然而，CLIP通常在联合嵌入空间中存在图像和文本模态对齐不佳的问题。我们研究了针对下游图像字幕生成任务高效地线性重新对齐联合嵌入空间的方法。这带来了一种高效的训练协议，仅需在联合CLIP空间中计算线性映射的闭式解。因此，我们提出了一种轻量级字幕生成方法，称为ReCap，其训练速度比现有轻量级方法快达1000倍。此外，我们基于CLIP分数以及我们提出的对齐方法，提出了两种新的基于学习的图像字幕评估指标。我们在MS-COCO、Flickr30k、VizWiz和MSRVTT上评估了ReCap。在前两个数据集上，ReCap在使用基于规则的指标时与最先进的轻量级方法表现相当，同时在大多数基于CLIP的指标上优于它们。在后两个基准上，ReCap在所有指标上一致优于竞争对手，并展现出强大的迁移能力和对噪声的鲁棒性。最后，我们证明，在Flickr8k-Expert、Flickr8k-Crowdflower和THumB数据集上，我们提出的指标与人类判断的相关性强于现有指标。

英文摘要：

Recently, vision-language models like CLIP have advanced the state of the art in a variety of multi-modal tasks including image captioning and caption evaluation. Many approaches leverage CLIP for cross-modal retrieval to condition pre-trained language models on visual input. However, CLIP generally suffers from a mis-alignment of image and text modalities in the joint embedding space. We investigate efficient methods to linearly re-align the joint embedding space for the downstream task of image captioning. This leads to an efficient training protocol that merely requires computing a closed-form solution for a linear mapping in the joint CLIP space. Consequently, we propose a lightweight captioning method called ReCap, which can be trained up to 1000 times faster than existing lightweight methods. Moreover, we propose two new learning-based image-captioning metrics built on CLIP score along with our proposed alignment. We evaluate ReCap on MS-COCO, Flickr30k, VizWiz and MSRVTT. On the former two, ReCap performs comparably to state-of-the-art lightweight methods using rule-based metrics while outperforming them on most of the CLIP-based metrics. On the latter two benchmarks, ReCap consistently outperforms competitors across all metrics and exhibits strong transfer capabilities and resilience to noise. Finally, we demonstrate that our proposed metrics correlate stronger with human judgement than existing metrics on the Flickr8k-Expert, Flickr8k-Crowdflower, and THumB datasets.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。