基于扩散模型的风格迁移用于合成到真实域自适应
Style Transfer with Diffusion Models for Synthetic-to-Real Domain Adaptation
AI总结：
本文提出基于扩散模型的类别感知风格迁移方法 CACTI 和 CACTIF，在保留语义边界的同时弥合合成到真实域差距，实验表明其能生成更高质量图像并提升分割模型性能。
摘要：
在合成数据上训练的语义分割模型由于域差距，在真实世界图像上通常表现不佳，尤其是在标注数据稀缺的恶劣条件下。然而，最近的 foundation models 使得无需任何训练即可生成逼真图像。本文提出利用此类扩散模型来提升在合成数据上学习的视觉模型的性能。我们引入了两种使用扩散模型进行语义一致风格迁移的新技术：类别自适应实例归一化与交叉注意力（CACTI）及其带选择性注意力过滤的扩展（CACTIF）。CACTI 基于语义类别选择性地应用统计归一化，而 CACTIF 进一步基于特征相似性过滤交叉注意力图，防止在交叉注意力对应较弱的区域产生伪影。与采用全局变换或无约束生成内容的方法不同，我们的方法在迁移风格特征的同时保留语义边界和结构一致性。使用 GTA5 作为源域、Cityscapes/ACDC 作为目标域的实验表明，我们的方法生成了更高质量的图像，具有更低的 FID 分数和更好的内容保留。我们的工作证明，即使在目标域数据极少的情况下，基于类别感知的扩散风格迁移也能有效弥合合成到真实的域差距，推动面向具有挑战性的真实世界应用的鲁棒感知系统发展。源代码可在以下地址获取：https://github.com/echigot/cactif。
英文摘要：
Semantic segmentation models trained on synthetic data often perform poorly on real-world images due to domain gaps, particularly in adverse conditions where labeled data is scarce. Yet, recent foundation models enable to generate realistic images without any training. This paper proposes to leverage such diffusion models to improve the performance of vision models when learned on synthetic data. We introduce two novel techniques for semantically consistent style transfer using diffusion models: Class-wise Adaptive Instance Normalization and Cross-Attention (CACTI) and its extension with selective attention Filtering (CACTIF). CACTI applies statistical normalization selectively based on semantic classes, while CACTIF further filters cross-attention maps based on feature similarity, preventing artifacts in regions with weak cross-attention correspondences. Our methods transfer style characteristics while preserving semantic boundaries and structural coherence, unlike approaches that apply global transformations or generate content without constraints. Experiments using GTA5 as source and Cityscapes/ACDC as target domains show that our approach produces higher quality images with lower FID scores and better content preservation. Our work demonstrates that class-aware diffusion-based style transfer effectively bridges the synthetic-to-real domain gap even with minimal target domain data, advancing robust perception systems for challenging real-world applications. The source code is available at: https://github.com/echigot/cactif.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
