ControlUDA：用于跨天气语义分割的可控扩散辅助无监督域适应
ControlUDA: Controllable Diffusion-assisted Unsupervised Domain Adaptation for Cross-Weather Semantic Segmentation
- Technical University of Munich（慕尼黑工业大学）
- Huawei Munich Research Center（华为慕尼黑研究中心）
- Peking University（北京大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对恶劣天气语义分割的无监督域适应，提出 ControlUDA 框架，利用扩散模型生成可控天气的高保真数据，通过目标先验和 UDAControlNet 解决标签缺失与生成质量问题，在 Cityscapes-to-ACDC 基准上达到 72.0 mIoU。
AI中文摘要：
数据生成被认为是针对恶劣天气下语义分割的无监督域适应（UDA）的一种有效策略。然而，这些恶劣天气场景包含多种可能性，且在以往的无监督域适应工作中，具有可控天气的高保真数据合成研究不足。近年来，大规模文本到图像扩散模型（DM）的进展为研究开辟了新途径，使得能够基于语义标签生成逼真图像。由于源域和目标域共享标签空间，这一能力对于从源域到目标域的跨域数据合成至关重要。因此，源域标签可以与生成的伪目标数据配对，用于训练无监督域适应模型。然而，从无监督域适应的角度来看，扩散模型训练存在几个挑战：（i）目标域的 ground-truth 标签缺失；（ii）提示生成器可能为恶劣天气图像生成模糊或嘈杂的描述；（iii）现有技术通常在仅基于语义标签的条件下，难以妥善处理城市场景的复杂场景结构和几何信息。为解决上述问题，我们提出了 ControlUDA，一个专为恶劣天气条件下无监督域适应分割设计的扩散辅助框架。它首先利用预训练分割器的目标先验来调整扩散模型，弥补目标域标签的缺失；它还包含 UDAControlNet，一个针对恶劣天气高保真数据生成的条件融合多尺度及提示增强网络。使用我们生成的数据训练无监督域适应模型，在流行的 Cityscapes-to-ACDC 恶劣天气基准上达到了新的里程碑（72.0 mIoU）。此外，ControlUDA 有助于在未见数据上实现良好的模型泛化能力。
英文摘要：
Data generation is recognized as a potent strategy for unsupervised domain adaptation (UDA) pertaining semantic segmentation in adverse weathers. Nevertheless, these adverse weather scenarios encompass multiple possibilities, and high-fidelity data synthesis with controllable weather is under-researched in previous UDA works. The recent strides in large-scale text-to-image diffusion models (DM) have ushered in a novel avenue for research, enabling the generation of realistic images conditioned on semantic labels. This capability proves instrumental for cross-domain data synthesis from source to target domain owing to their shared label space. Thus, source domain labels can be paired with those generated pseudo target data for training UDA. However, from the UDA perspective, there exists several challenges for DM training: (i) ground-truth labels from target domain are missing; (ii) the prompt generator may produce vague or noisy descriptions of images from adverse weathers; (iii) existing arts often struggle to well handle the complex scene structure and geometry of urban scenes when conditioned only on semantic labels. To tackle the above issues, we propose ControlUDA, a diffusion-assisted framework tailored for UDA segmentation under adverse weather conditions. It first leverages target prior from a pre-trained segmentor for tuning the DM, compensating the missing target domain labels; It also contains UDAControlNet, a condition-fused multi-scale and prompt-enhanced network targeted at high-fidelity data generation in adverse weathers. Training UDA with our generated data brings the model performances to a new milestone (72.0 mIoU) on the popular Cityscapes-to-ACDC benchmark for adverse weathers. Furthermore, ControlUDA helps to achieve good model generalizability on unseen data.
