JoDiffusion: 通过像素级标注联合扩散图像以促进语义分割
JoDiffusion: Jointly Diffusing Image with Pixel-Level Annotations for Semantic Segmentation Promotion
AI总结：
JoDiffusion通过联合扩散图像与像素级标注，提升语义分割的性能和可扩展性。
摘要：
鉴于像素级标注固有地成本高且耗时，生成包含足够多样合成图像及其真实像素级标注的合成数据集最近受到越来越多的关注，用于训练高性能的语义分割模型。然而，现有方法需要在图像生成后预测伪标注或根据手动标注掩码生成图像，这导致了图像-标注语义不一致或可扩展性问题。为一劳永逸地解决这两个问题，我们提出了一种新的语义分割数据生成扩散框架，称为JoDiffusion。首先，给定一个标准的潜在扩散模型，JoDiffusion结合了一个独立的标注变分自编码器（VAE）网络，将标注掩码映射到与图像共享的潜在空间。然后，扩散模型被调整以捕捉图像及其标注掩码在文本提示条件下的联合分布。通过这样做，JoDiffusion能够同时生成配对的图像和语义一致的标注掩码，仅根据文本提示进行条件生成，从而展示出优越的可扩展性。此外，开发了一种掩码优化策略以减轻生成过程中产生的标注噪声。在Pascal VOC、COCO和ADE20K数据集上的实验表明，由JoDiffusion生成的标注数据集在语义分割中相比现有方法产生了显著的性能提升。
英文摘要：
Given the inherently costly and time-intensive nature of pixel-level annotation, the generation of synthetic datasets comprising sufficiently diverse synthetic images paired with ground-truth pixel-level annotations has garnered increasing attention recently for training high-performance semantic segmentation models. However, existing methods necessitate to either predict pseudo annotations after image generation or generate images conditioned on manual annotation masks, which incurs image-annotation semantic inconsistency or scalability problem. To migrate both problems with one stone, we present a novel dataset generative diffusion framework for semantic segmentation, termed JoDiffusion. Firstly, given a standard latent diffusion model, JoDiffusion incorporates an independent annotation variational auto-encoder (VAE) network to map annotation masks into the latent space shared by images. Then, the diffusion model is tailored to capture the joint distribution of each image and its annotation mask conditioned on a text prompt. By doing these, JoDiffusion enables simultaneously generating paired images and semantically consistent annotation masks solely conditioned on text prompts, thereby demonstrating superior scalability. Additionally, a mask optimization strategy is developed to mitigate the annotation noise produced during generation. Experiments on Pascal VOC, COCO, and ADE20K datasets show that the annotated dataset generated by JoDiffusion yields substantial performance improvements in semantic segmentation compared to existing methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
