通过在推理时模仿人类改写反馈来改进图像描述生成
Improving Image Captioning by Mimicking Human Reformulation Feedback at Inference-time
AI总结：
本文提出在推理阶段利用人工描述改写反馈训练改写模型，以修正图像描述错误并实现风格迁移，在德语描述和英语风格迁移上达到SOTA。
摘要：
将自动预测的人类反馈纳入生成模型的训练过程，近年来引起了广泛关注，而推理阶段的反馈则受到较少关注。典型的训练时反馈，即在给定两个样本时的选择偏好，并不能自然迁移到推理阶段。我们引入了一种新型反馈——描述改写（caption reformulations），并训练模型基于人工标注来模仿改写反馈。我们的方法不需要训练图像描述模型本身，因此所需的计算量大幅减少。我们实验了两类改写反馈：首先，我们收集了一个人工改写数据集，用于纠正生成描述中的错误。我们发现，将在此数据上训练的改写模型纳入现有图像描述模型的推理阶段，能够改进描述质量，尤其是在原始描述质量较低时。我们将该方法应用于非英语图像描述这一鲁棒模型相对稀缺的领域，并获得了显著提升。其次，我们将改写应用于风格迁移。定量评估显示，该方法在德语图像描述和英语风格迁移任务上达到了最先进性能；同时，采用详细比较框架进行的人工验证揭示了具体的改进维度。
英文摘要：
Incorporating automatically predicted human feedback into the process of training generative models has attracted substantial recent interest, while feedback at inference time has received less attention. The typical feedback at training time, i.e., preferences of choice given two samples, does not naturally transfer to the inference phase. We introduce a novel type of feedback -- caption reformulations -- and train models to mimic reformulation feedback based on human annotations. Our method does not require training the image captioning model itself, thereby demanding substantially less computational effort. We experiment with two types of reformulation feedback: first, we collect a dataset of human reformulations that correct errors in the generated captions. We find that incorporating reformulation models trained on this data into the inference phase of existing image captioning models results in improved captions, especially when the original captions are of low quality. We apply our method to non-English image captioning, a domain where robust models are less prevalent, and gain substantial improvement. Second, we apply reformulations to style transfer. Quantitative evaluations reveal state-of-the-art performance on German image captioning and English style transfer, while human validation with a detailed comparative framework exposes the specific axes of improvement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
