将人类纳入图像描述生成循环
Putting Humans in the Image Captioning Loop
AI总结：
本文提出在MS COCO预训练图像描述模型中引入用户反馈，通过数据增强、逐步更新与稀疏记忆回放实现个性化适配并缓解灾难性遗忘。
摘要：
图像描述生成（Image Captioning，IC）模型在训练过程中可从人类反馈中大幅受益，尤其是在数据有限的情况下。我们展示了一项正在进行的工作：调整一个IC系统以整合人类反馈，目标是使其能够轻松适配用户特定数据。我们的方法建立在一个基于MS COCO数据集预训练的基础IC模型之上，该模型为未见过的图像生成描述。随后，用户可以针对图像以及生成/预测的描述提供反馈，这些反馈将被增强，以创建额外的训练实例，用于模型适配。额外实例通过逐步更新被整合到模型中，并使用稀疏记忆回放组件来避免灾难性遗忘。我们希望该方法在带来更好结果的同时，也能产生可定制的IC模型。
英文摘要：
Image Captioning (IC) models can highly benefit from human feedback in the training process, especially in cases where data is limited. We present work-in-progress on adapting an IC system to integrate human feedback, with the goal to make it easily adaptable to user-specific data. Our approach builds on a base IC model pre-trained on the MS COCO dataset, which generates captions for unseen images. The user will then be able to offer feedback on the image and the generated/predicted caption, which will be augmented to create additional training instances for the adaptation of the model. The additional instances are integrated into the model using step-wise updates, and a sparse memory replay component is used to avoid catastrophic forgetting. We hope that this approach, while leading to improved results, will also result in customizable IC models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
