ECA：面向开放图像到文本生成的高效持续对齐
ECA: Efficient Continual Alignment for Open-Ended Image-to-Text Generation
AI总结：
提出ECA方法，通过混合查询模块、Fisher动态扩展和字典重放，实现无需旧数据的持续对齐，缓解灾难性遗忘，提升开放图像到文本生成的增量学习性能。
摘要：
开放图像到文本生成（OpenITG）的增量学习（IL）使模型能够持续为新的图像生成准确、上下文相关的文本，同时保留先前获得的知识。与先前研究不同，本文处理了一个更实际的场景，其中视觉数据的主要类别随时间推移而演变。在此背景下，我们引入了持续对齐的新概念，它逐步调整预训练VLM中的对齐模块，以保持高质量的跨模态表示。基于这一思想，我们提出了高效持续对齐（ECA），一种用于OpenITG的无样本IL方法。关键挑战是使模型能够获取新的任务特定特征，同时最小化对已建立对齐的干扰，且无需访问先前任务的原始数据。为此，ECA采用了三种核心机制：混合查询（MoQ）模块，用于适应任务特定的查询令牌；Fisher动态扩展（FeDEx），基于Fisher信息矩阵（FIM）度量动态扩展模型结构；以及带有字典重放（DR）的嵌入字典，以保留过去的知识。为了评估ECA的性能，我们构建了四个新的IL OpenITG基准，更好地反映了现实场景。实验结果表明，与基线方法相比，ECA显著缓解了灾难性遗忘并提高了IL性能。代码和基准可在该https URL获取。
英文摘要：
Incremental Learning (IL) for Open-ended Image-to-Text Generation (OpenITG) enables models to continuously generate accurate, contextually relevant text for new images while preserving previously acquired knowledge. Unlike prior studies, this paper addresses a more practical scenario in which the predominant category of visual data shifts over time as environments evolve. In this context, we introduce a new notion of continual alignment, which incrementally adapts the alignment module within pre-trained VLMs to preserve high-quality cross-modal representations. Based on this idea, we propose Efficient Continual Alignment (ECA), a novel exemplar-free IL approach for OpenITG. The key challenge is enabling the model to acquire new, task-specific features while minimizing interference with the established alignment without accessing raw data from previous tasks. To address this, ECA employs three core mechanisms: a Mixture of Query (MoQ) module that adapts task-specific query tokens, a Fisher Dynamic Expansion (FeDEx) that dynamically expands model structure based on a Fisher Information Matrix (FIM)-based metric, and an embedding dictionary with Dictionary Replay (DR) to retain past knowledge. To evaluate ECA's performance, we construct four new IL OpenITG benchmarks that better reflect real-world scenarios. Experimental results demonstrate that ECA significantly mitigates catastrophic forgetting and improves IL performance compared to baseline methods. Code and benchmarks are available at https://github.com/Snowball0823/ECA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
