arXiv 2308.011272023-08-02cs.CV

DiffusePast：用于类别增量语义分割的基于扩散的生成式重放

DiffusePast: Diffusion-based Generative Replay for Class Incremental Semantic Segmentation

The Hong Kong Polytechnic University（香港理工大学）；Center for Artificial Intelligence and Robotics, HKISI, CAS（中国科学院香港创新研究院人工智能与机器人创新中心）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）

Jingfan Chen, Yuxi Wang, Pengfei Wang, Xiao Chen, Zhaoxiang Zhang, Zhen Lei, Qing Li

备注：: e.g.: 13 pages, 7 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出DiffusePast框架，通过基于扩散模型的双生成器范式生成语义准确的旧类图像及可靠掩码，结合逐类令牌嵌入和背景伪标签分配，有效缓解了类别增量语义分割中的灾难性遗忘问题。

摘要：

类别增量语义分割（CISS）通过增量学习新增类别来扩展传统的分割任务。先前的工作引入了生成式重放，即重放由预训练的GAN生成的旧类样本，以解决灾难性遗忘和隐私问题。然而，生成的图像缺乏语义精度并表现出分布外特征，导致掩码不准确，进一步降低了分割性能。为了应对这些挑战，我们提出了DiffusePast，这是一个新颖的框架，其特点是基于扩散的生成式重放模块，该模块在不同指令（例如文本提示或边缘图）的引导下生成语义准确的图像和更可靠的掩码。具体而言，DiffusePast引入了双生成器范式，专注于生成与下游数据集分布对齐的旧类图像，同时保留原始图像的结构和布局，从而实现更精确的掩码。为了持续适应新增类别的新视觉概念，我们在更新双生成器时结合了逐类令牌嵌入。此外，我们在新步骤图像中为背景像素分配了充足的旧类伪标签，进一步缓解了对先前所学知识的遗忘。通过全面的实验，我们的方法在主流基准上展现了有竞争力的性能，在旧类和新类的性能之间取得了更好的平衡。

英文摘要：

The Class Incremental Semantic Segmentation (CISS) extends the traditional segmentation task by incrementally learning newly added classes. Previous work has introduced generative replay, which involves replaying old class samples generated from a pre-trained GAN, to address the issues of catastrophic forgetting and privacy concerns. However, the generated images lack semantic precision and exhibit out-of-distribution characteristics, resulting in inaccurate masks that further degrade the segmentation performance. To tackle these challenges, we propose DiffusePast, a novel framework featuring a diffusion-based generative replay module that generates semantically accurate images with more reliable masks guided by different instructions (e.g., text prompts or edge maps). Specifically, DiffusePast introduces a dual-generator paradigm, which focuses on generating old class images that align with the distribution of downstream datasets while preserving the structure and layout of the original images, enabling more precise masks. To adapt to the novel visual concepts of newly added classes continuously, we incorporate class-wise token embedding when updating the dual-generator. Moreover, we assign adequate pseudo-labels of old classes to the background pixels in the new step images, further mitigating the forgetting of previously learned knowledge. Through comprehensive experiments, our method demonstrates competitive performance across mainstream benchmarks, striking a better balance between the performance of old and novel classes.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。