通过基于CLIP的直接优化重新审视图像描述训练范式
Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimization
AI总结：
本文提出Direct CLIP-Based Optimization (DiCO)新训练范式，通过联合优化蒸馏的奖励模型并解决加权分类问题，提升图像描述的稳定性和质量，更符合人类偏好。
摘要：
图像描述的传统训练方法包括使用teacher forcing预训练网络，随后使用自临界序列训练进行微调，以最大化手工制作的描述指标。然而，当试图优化CLIP-Score和PAC-Score等现代且更高质量的指标时，这种训练方法经常遇到不稳定性，并且无法获得生成流畅且信息丰富的描述所需的真正描述能力。在本文中，我们提出了一种称为Direct CLIP-Based Optimization (DiCO)的新训练范式。我们的方法联合学习和优化一个从具有高人类相关性的可学习描述评估器中蒸馏出的奖励模型。这是通过直接在描述器内部解决加权分类问题来完成的。同时，DiCO防止与原始模型发散，确保保持流畅性。DiCO不仅在生成的描述中表现出改进的稳定性和增强的质量，而且与现有方法相比更符合人类偏好，特别是在现代指标上。此外，它在传统指标上保持有竞争力的性能。我们的源代码和训练模型公开在https://github.com/aimagelab/DiCO。
英文摘要：
The conventional training approach for image captioning involves pre-training a network using teacher forcing and subsequent fine-tuning with Self-Critical Sequence Training to maximize hand-crafted captioning metrics. However, when attempting to optimize modern and higher-quality metrics like CLIP-Score and PAC-Score, this training method often encounters instability and fails to acquire the genuine descriptive capabilities needed to produce fluent and informative captions. In this paper, we propose a new training paradigm termed Direct CLIP-Based Optimization (DiCO). Our approach jointly learns and optimizes a reward model that is distilled from a learnable captioning evaluator with high human correlation. This is done by solving a weighted classification problem directly inside the captioner. At the same time, DiCO prevents divergence from the original model, ensuring that fluency is maintained. DiCO not only exhibits improved stability and enhanced quality in the generated captions but also aligns more closely with human preferences compared to existing methods, especially in modern metrics. Additionally, it maintains competitive performance in traditional metrics. Our source code and trained models are publicly available at https://github.com/aimagelab/DiCO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
