基于改进图文对齐的图像描述持续学习方法
Continual Learning for Image Captioning through Improved Image-Text Alignment
AI总结：
针对持续图像描述中的灾难性遗忘与图文对齐难题，提出融合三类对齐损失的多损失框架，在无推理额外开销的前提下，有效缓解遗忘并取得优于现有方法的语义对齐效果。
摘要：
在持续学习场景下生成准确连贯的图像描述仍是一项重大挑战，其根源在于灾难性遗忘问题，以及随时间推移将不断演化的视觉概念与语言对齐的难度。本研究提出一种面向持续图像描述任务的新型多损失框架，该框架通过基于提示的持续学习与对比对齐融合了语义引导机制。该方法以预训练的ViT-GPT-2为主干网络，在标准交叉熵损失之外结合了三个额外损失项：（1）基于提示的余弦相似度损失，将图像嵌入与编码了物体、属性和动作的合成构建提示进行对齐；（2）CLIP风格损失，用于促进图像嵌入与目标描述嵌入之间的对齐；（3）语言引导对比损失，采用三元组损失增强跨任务的类别级可区分性。值得注意的是，该方法在推理阶段不会引入额外开销，且在描述生成过程中无需使用提示。实验结果表明，该方法可缓解灾难性遗忘问题，同时与当前最优方法相比实现了更优的描述语义对齐效果。项目代码可通过链接https://github.com/Gepardius/Taetz_Bordelius_Continual_ImageCaptioning获取。
英文摘要：
Generating accurate and coherent image captions in a continual learning setting remains a major challenge due to catastrophic forgetting and the difficulty of aligning evolving visual concepts with language over time. In this work, we propose a novel multi-loss framework for continual image captioning that integrates semantic guidance through prompt-based continual learning and contrastive alignment. Built upon a pretrained ViT-GPT-2 backbone, our approach combines standard cross-entropy loss with three additional components: (1) a prompt-based cosine similarity loss that aligns image embeddings with synthetically constructed prompts encoding objects, attributes, and actions; (2) a CLIP-style loss that promotes alignment between image embeddings and target caption embedding; and (3) a language-guided contrastive loss that employs a triplet loss to enhance class-level discriminability between tasks. Notably, our approach introduces no additional overhead at inference time and requires no prompts during caption generation. We find that this approach mitigates catastrophic forgetting, while achieving better semantic caption alignment compared to state-of-the-art methods. The code can be found via the following link: https://github.com/Gepardius/Taetz_Bordelius_Continual_ImageCaptioning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
