arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.04295cs.CV
收藏

一切皆合成:通过合成域对齐的扩散驱动测试时自适应

Everything to the Synthetic: Diffusion-driven Test-time Adaptation via Synthetic-Domain Alignment

Tsinghua University(清华大学);Georgia Tech(佐治亚理工学院)

Jiayi Guo, Junhao Zhao, Chaoqun Du, Yulin Wang, Chunjiang Ge, Zanlin Ni, Shiji Song, Humphrey Shi, Gao Huang

AI总结:

针对扩散驱动测试时自适应中合成数据与源数据域不对齐的问题,提出SDA框架,通过条件与无条件扩散模型混合生成数据微调源模型,在分类、分割及多模态大语言模型上优于现有方法。

摘要:

测试时自适应(TTA)旨在提升源域预训练模型在未见过的、发生偏移的目标域上的性能。传统的TTA方法主要基于目标数据流调整模型权重,这使得模型性能对目标数据的数量和顺序敏感。最近提出的扩散驱动TTA方法通过调整模型输入而非权重来缓解这一问题,其中在源域上训练的无条件扩散模型将目标域数据转换为预期接近源域的合成域。然而,在本文中,我们揭示尽管扩散驱动TTA中的合成数据与源数据在视觉上难以区分,但对于深度网络而言,它们与源数据不对齐,甚至显著不同。为解决这一问题,我们提出了一个合成域对齐(SDA)框架。我们的关键见解是利用合成数据微调源模型以确保更好的对齐。具体而言,我们首先使用条件扩散模型生成带标签的样本,创建合成数据集。随后,在微调之前,我们使用前述无条件扩散模型对每个样本进行加噪和去噪。这种扩散混合(MoD)过程缓解了条件模型与无条件模型之间潜在的域不对齐问题。在分类器、分割器以及多模态大语言模型(MLLMs,例如LLaVA)上的大量实验表明,SDA实现了优越的域对齐,并持续优于现有的扩散驱动TTA方法。我们的代码可在https://github.com/SHI-Labs/Diffusion-Driven-Test-Time-Adaptation-via-Synthetic-Domain-Alignment获取。

英文摘要:

Test-time adaptation (TTA) aims to improve the performance of source-domain pre-trained models on previously unseen, shifted target domains. Traditional TTA methods primarily adapt model weights based on target data streams, making model performance sensitive to the amount and order of target data. The recently proposed diffusion-driven TTA methods mitigate this by adapting model inputs instead of weights, where an unconditional diffusion model, trained on the source domain, transforms target-domain data into a synthetic domain that is expected to approximate the source domain. However, in this paper, we reveal that although the synthetic data in diffusion-driven TTA seems indistinguishable from the source data, it is unaligned with, or even markedly different from the latter for deep networks. To address this issue, we propose a \textbf{S}ynthetic-\textbf{D}omain \textbf{A}lignment (SDA) framework. Our key insight is to fine-tune the source model with synthetic data to ensure better alignment. Specifically, we first employ a conditional diffusion model to generate labeled samples, creating a synthetic dataset. Subsequently, we use the aforementioned unconditional diffusion model to add noise to and denoise each sample before fine-tuning. This Mix of Diffusion (MoD) process mitigates the potential domain misalignment between the conditional and unconditional models. Extensive experiments across classifiers, segmenters, and multimodal large language models (MLLMs, \eg, LLaVA) demonstrate that SDA achieves superior domain alignment and consistently outperforms existing diffusion-driven TTA methods. Our code is available at https://github.com/SHI-Labs/Diffusion-Driven-Test-Time-Adaptation-via-Synthetic-Domain-Alignment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑