Bisecle：视频语言理解持续学习中的绑定与分离
Bisecle: Binding and Separation in Continual Learning for Video Language Understanding
AI总结：
本文受海马体绑定与分离机制启发，提出Bisecle方法，通过多方向监督和对比提示学习解决视频-语言持续学习中的灾难性遗忘与更新冲突问题。
摘要：
前沿视觉-语言模型（VLMs）在视频理解任务中取得了显著进展。然而，现实世界中的视频通常以持续演化的数据流形式存在（例如，可穿戴眼镜捕捉的动态场景），这要求模型能够持续适应不断变化的数据分布和新颖场景。考虑到在新任务上微调模型所带来的高昂计算成本，通常只更新一小部分参数，而模型主体保持冻结。这在大型多模态基础模型的背景下，给现有的持续学习框架带来了新的挑战，即灾难性遗忘和更新冲突。虽然基础模型在参数高效的持续学习方面面临困难，但人脑中的海马体已经进化出高效的记忆形成与巩固机制。受海马体中快速绑定和模式分离机制的启发，本文提出了用于视频-语言持续学习的Bisecle方法，其中使用多方向监督模块来捕获更多的跨模态关系，并设计了一种对比提示学习方案来隔离任务特定知识，以促进高效的记忆存储。绑定和分离过程进一步增强了VLMs保留复杂经验的能力，从而在视频理解任务中实现稳健且高效的持续学习。我们对所提出的Bisecle进行了全面评估，在多个VideoQA基准上证明了其缓解遗忘和增强跨任务泛化的能力。
英文摘要：
Frontier vision-language models (VLMs) have made remarkable improvements in video understanding tasks. However, real-world videos typically exist as continuously evolving data streams (e.g., dynamic scenes captured by wearable glasses), necessitating models to continually adapt to shifting data distributions and novel scenarios. Considering the prohibitive computational costs of fine-tuning models on new tasks, usually, a small subset of parameters is updated while the bulk of the model remains frozen. This poses new challenges to existing continual learning frameworks in the context of large multimodal foundation models, i.e., catastrophic forgetting and update conflict. While the foundation models struggle with parameter-efficient continual learning, the hippocampus in the human brain has evolved highly efficient mechanisms for memory formation and consolidation. Inspired by the rapid Binding and pattern separation mechanisms in the hippocampus, in this work, we propose Bisecle for video-language continual learning, where a multi-directional supervision module is used to capture more cross-modal relationships and a contrastive prompt learning scheme is designed to isolate task-specific knowledge to facilitate efficient memory storage. Binding and separation processes further strengthen the ability of VLMs to retain complex experiences, enabling robust and efficient continual learning in video understanding tasks. We perform a thorough evaluation of the proposed Bisecle, demonstrating its ability to mitigate forgetting and enhance cross-task generalization on several VideoQA benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
