MCF-VC：缓解多模态视频字幕生成中类增量学习的灾难性遗忘
MCF-VC: Mitigate Catastrophic Forgetting in Class-Incremental Learning for Multimodal Video Captioning
AI总结：
本文提出 MCF-VC 方法，通过细粒度敏感性选择与两阶段知识蒸馏缓解多模态视频字幕生成中的类增量灾难性遗忘，并在 MSR-VTT 上验证其有效性。
摘要：
为了解决顺序输入中旧类别不可见所导致的灾难性遗忘问题，现有基于相对简单分类任务的工作已经取得了一定进展。相比之下，视频字幕生成是多模态场景中更复杂的任务，尚未在增量学习领域得到探索。在分析顺序输入视频时识别出这一稳定性-可塑性问题后，我们原创性地提出了一种在多模态视频字幕生成的类增量学习中缓解灾难性遗忘的方法（MCF-VC）。为了在宏观层面有效保持旧任务上的良好性能，我们基于线性层参数掩码和 Fisher 敏感性设计了细粒度敏感性选择（FgSS），以从旧任务中挑选有用知识。进一步，为了在具体特征层面更好地约束新旧任务的知识特性，我们构建了两阶段知识蒸馏（TsKD），能够在权衡旧任务的同时学好新任务。具体而言，我们设计了两个蒸馏损失，分别约束语义注意力特征图的跨模态语义信息和最终输出的文本信息，从而在学习新类的同时保留旧类的模型间和模型内风格化知识。为了说明我们的模型抵抗遗忘的能力，我们设计了一个指标 CIDER_t 来检测阶段遗忘率。我们在公开数据集 MSR-VTT 上的实验表明，所提出的方法在不重放旧样本的情况下显著抵抗了对先前任务的遗忘，并且在新任务上表现良好。
英文摘要：
To address the problem of catastrophic forgetting due to the invisibility of old categories in sequential input, existing work based on relatively simple categorization tasks has made some progress. In contrast, video captioning is a more complex task in multimodal scenario, which has not been explored in the field of incremental learning. After identifying this stability-plasticity problem when analyzing video with sequential input, we originally propose a method to Mitigate Catastrophic Forgetting in class-incremental learning for multimodal Video Captioning (MCF-VC). As for effectively maintaining good performance on old tasks at the macro level, we design Fine-grained Sensitivity Selection (FgSS) based on the Mask of Linear's Parameters and Fisher Sensitivity to pick useful knowledge from old tasks. Further, in order to better constrain the knowledge characteristics of old and new tasks at the specific feature level, we have created the Two-stage Knowledge Distillation (TsKD), which is able to learn the new task well while weighing the old task. Specifically, we design two distillation losses, which constrain the cross modal semantic information of semantic attention feature map and the textual information of the final outputs respectively, so that the inter-model and intra-model stylized knowledge of the old class is retained while learning the new class. In order to illustrate the ability of our model to resist forgetting, we designed a metric CIDER_t to detect the stage forgetting rate. Our experiments on the public dataset MSR-VTT show that the proposed method significantly resists the forgetting of previous tasks without replaying old samples, and performs well on the new task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
