ConSept：基于适配器的视觉Transformer实现持续语义分割
ConSept: Continual Semantic Segmentation via Adapter-based Vision Transformer
AI总结：
本文提出ConSept方法，通过在普通ViT中集成轻量级注意力适配器，并结合确定性旧类别边界蒸馏与双重dice损失，有效缓解持续语义分割中的灾难性遗忘问题，同时提升新旧类别的分割性能。
摘要：
本文深入探讨了视觉Transformer在持续语义分割中的应用，该问题在以往文献中尚未得到充分研究。通过对现有框架适配到普通ViT的实证研究发现，在ViT中引入视觉适配器或使用蒸馏项微调ViT，有利于增强新类别的分割能力。这些发现促使我们提出基于适配器ViT的持续语义分割方法，即ConSept。在带有线性分割头的简化ViT架构中，ConSept将轻量级的基于注意力的适配器集成到普通ViT中。利用这些适配器的特征适应能力，ConSept不仅对旧类别保持了优越的分割能力，而且对新类别也取得了良好的分割质量。为了进一步发挥ConSept固有的抗灾难性遗忘能力，并同时提升新旧类别的分割性能，我们提出了两个关键策略：使用确定性旧类别边界的蒸馏以改善抗灾难性遗忘，以及双重dice损失来正则化分割图，从而提升整体分割性能。大量实验表明，ConSept在重叠或不重叠设置下的多个持续语义分割基准上均具有有效性。代码将在\url{https://github.com/DongSky/ConSept}公开。
英文摘要：
In this paper, we delve into the realm of vision transformers for continual semantic segmentation, a problem that has not been sufficiently explored in previous literature. Empirical investigations on the adaptation of existing frameworks to vanilla ViT reveal that incorporating visual adapters into ViTs or fine-tuning ViTs with distillation terms is advantageous for enhancing the segmentation capability of novel classes. These findings motivate us to propose Continual semantic Segmentation via Adapter-based ViT, namely ConSept. Within the simplified architecture of ViT with linear segmentation head, ConSept integrates lightweight attention-based adapters into vanilla ViTs. Capitalizing on the feature adaptation abilities of these adapters, ConSept not only retains superior segmentation ability for old classes, but also attains promising segmentation quality for novel classes. To further harness the intrinsic anti-catastrophic forgetting ability of ConSept and concurrently enhance the segmentation capabilities for both old and new classes, we propose two key strategies: distillation with a deterministic old-classes boundary for improved anti-catastrophic forgetting, and dual dice losses to regularize segmentation maps, thereby improving overall segmentation performance. Extensive experiments show the effectiveness of ConSept on multiple continual semantic segmentation benchmarks under overlapped or disjoint settings. Code will be publicly available at \url{https://github.com/DongSky/ConSept}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
