魔鬼在EOS中：面向详细图像描述的序列训练
The Devil is in the EOS: Sequence Training for Detailed Image Captioning
AI总结：
本文提出一种无监督方法消除交叉熵训练中模型对EOS标记的偏置，从而在不依赖复杂奖励或监督的情况下生成更长、更详细的图像描述，并在多个VLMs和基准上验证了有效性。
摘要：
尽管视觉-语言模型（VLMs）取得了显著进展，图像描述往往仍缺乏细节，基础模型生成的描述简短且泛化。即使VLMs配备了强大的视觉和语言骨干，这一局限依然存在。虽然已有监督数据和复杂奖励函数被提出用于改进详细图像描述，我们发现了一个更简单的潜在问题：在交叉熵训练中引入的对序列结束（EOS）标记的偏置。我们提出一种无监督方法来消除模型过早预测EOS标记的倾向。通过减少这种偏置，我们促使模型生成更长、更详细的描述，而无需复杂的奖励函数或监督。我们的方法简单、有效，并且易于应用于任何预训练模型。我们通过在三个VLMs和三个详细描述基准上的实验证明了其有效性。结果表明，描述长度和相关细节显著增加，尽管幻觉率如预期有所上升。
英文摘要：
Despite significant advances in vision-language models (VLMs), image captioning often suffers from a lack of detail, with base models producing short, generic captions. This limitation persists even though VLMs are equipped with strong vision and language backbones. While supervised data and complex reward functions have been proposed to improve detailed image captioning, we identify a simpler underlying issue: a bias towards the end-of-sequence (EOS) token, which is introduced during cross-entropy training. We propose an unsupervised method to debias the model's tendency to predict the EOS token prematurely. By reducing this bias, we encourage the generation of longer, more detailed captions without the need for intricate reward functions or supervision. Our approach is straightforward, effective, and easily applicable to any pretrained model. We demonstrate its effectiveness through experiments with three VLMs and on three detailed captioning benchmarks. Our results show a substantial increase in caption length and relevant details, albeit with an expected increase in the rate of hallucinations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
