arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.05005cs.CVcs.LGcs.RO
收藏

Diff-2-in-1:通过扩散模型弥合生成与密集感知的鸿沟

Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Carnegie Mellon University(卡内基梅隆大学);Tsinghua University(清华大学)

Shuhong Zheng, Zhipeng Bao, Ruoyu Zhao, Martial Hebert, Yu-Xiong Wang

AI总结:

Diff-2-in-1通过结合扩散模型的生成与感知能力,实现多模态数据生成与密集视觉感知的统一框架,提升视觉感知的判别能力。

摘要:

在高保真图像合成之外,扩散模型最近在密集视觉感知任务中展现出有希望的结果。然而,大多数现有工作将扩散模型视为感知任务中的独立组件,将其要么单独用于现成的数据增强,要么仅作为特征提取器。与这些孤立且不优的尝试相反,我们引入了一个统一、多功能的扩散基框架Diff-2-in-1,通过独特地利用扩散去噪过程,同时处理多模态数据生成和密集视觉感知。在此框架中,我们进一步通过利用去噪网络生成多模态数据来增强判别视觉感知,这些数据反映了原始训练集的分布。重要的是,Diff-2-in-1通过利用一种新的自我改进学习机制来优化所创建的多样且忠实的数据的利用。全面的实验评估验证了我们框架的有效性,展示了在各种判别后端和高质量多模态数据生成方面的一致性能提升,这些数据在真实性和实用性上都表现出色。

英文摘要:

Beyond high-fidelity image synthesis, diffusion models have recently exhibited promising results in dense visual perception tasks. However, most existing work treats diffusion models as a standalone component for perception tasks, employing them either solely for off-the-shelf data augmentation or as mere feature extractors. In contrast to these isolated and thus sub-optimal efforts, we introduce a unified, versatile, diffusion-based framework, Diff-2-in-1, that can simultaneously handle both multi-modal data generation and dense visual perception, through a unique exploitation of the diffusion-denoising process. Within this framework, we further enhance discriminative visual perception via multi-modal generation, by utilizing the denoising network to create multi-modal data that mirror the distribution of the original training set. Importantly, Diff-2-in-1 optimizes the utilization of the created diverse and faithful data by leveraging a novel self-improving learning mechanism. Comprehensive experimental evaluations validate the effectiveness of our framework, showcasing consistent performance improvements across various discriminative backbones and high-quality multi-modal data generation characterized by both realism and usefulness. Our project website is available at https://zsh2000.github.io/diff-2-in-1.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑