引导图像描述模型生成更具特异性的描述
Guiding Image Captioning Models Toward More Specific Captions
AI总结：
本文提出利用无分类器引导和语言模型引导自回归图像描述模型，在最少改动训练过程的前提下生成更具特异性的描述，显著提升无参考指标但略微降低基于参考的指标。
摘要：
图像描述通常被表述为生成与参考图像-描述对分布相匹配的图像描述的任务。然而，标准描述数据集中的参考描述很短，可能无法唯一标识它们所描述的图像。当模型直接在从互联网收集的图像-alt文本对上进行训练时，这些问题会进一步加剧。在这项工作中，我们表明可以通过对训练过程进行最少的更改来生成更具特异性的描述。我们通过微调自回归描述模型来估计描述上的条件分布和无条件分布，从而实现了无分类器引导。在解码时应用的引导尺度控制着最大化 $p(\mathrm{caption}|\mathrm{image})$ 和 $p(\mathrm{image}|\mathrm{caption})$ 之间的权衡。与标准贪婪解码相比，使用引导尺度为2的解码显著改善了无参考指标，如 CLIPScore（0.808 vs. 0.775）以及 CLIP 嵌入空间中的描述$\to$图像检索性能（recall@1 44.6% vs. 26.5%），但恶化了标准的基于参考的描述指标（例如，CIDEr 78.6 vs 126.1）。我们进一步探索了使用语言模型来引导解码过程，在由无分类器引导产生的无参考与基于参考的描述指标的 Pareto 前沿上获得了小幅改进，并大幅提升了仅在最少整理的网络数据上训练的模型所生成的描述质量。
英文摘要：
Image captioning is conventionally formulated as the task of generating captions for images that match the distribution of reference image-caption pairs. However, reference captions in standard captioning datasets are short and may not uniquely identify the images they describe. These problems are further exacerbated when models are trained directly on image-alt text pairs collected from the internet. In this work, we show that it is possible to generate more specific captions with minimal changes to the training process. We implement classifier-free guidance for an autoregressive captioning model by fine-tuning it to estimate both conditional and unconditional distributions over captions. The guidance scale applied at decoding controls a trade-off between maximizing $p(\mathrm{caption}|\mathrm{image})$ and $p(\mathrm{image}|\mathrm{caption})$. Compared to standard greedy decoding, decoding with a guidance scale of 2 substantially improves reference-free metrics such as CLIPScore (0.808 vs. 0.775) and caption$\to$image retrieval performance in the CLIP embedding space (recall@1 44.6% vs. 26.5%), but worsens standard reference-based captioning metrics (e.g., CIDEr 78.6 vs 126.1). We further explore the use of language models to guide the decoding process, obtaining small improvements over the Pareto frontier of reference-free vs. reference-based captioning metrics that arises from classifier-free guidance, and substantially improving the quality of captions generated from a model trained only on minimally curated web data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
