基于扩散变换器和CLIP的图像到图像翻译
Image-to-Image Translation with Diffusion Transformers and CLIP-Based Image Conditioning
AI总结：
本文提出基于扩散变换器和CLIP的图像到图像翻译方法，通过CLIP嵌入引导实现高质量、语义一致的图像转换，为配对图像翻译任务提供新方案。
摘要：
图像到图像翻译旨在学习源域和目标域之间的映射，从而实现风格迁移、外观转换和领域适应等任务。在本工作中，我们探索了一种基于扩散的框架用于图像到图像翻译，通过适应扩散变换器（DiT），将扩散模型的去噪能力与变换器的全局建模能力相结合。为了引导翻译过程，我们利用预训练CLIP编码器提取的图像嵌入来引导模型，从而实现细粒度且结构一致的翻译，而无需依赖文本或类别标签。我们结合了CLIP相似性损失以强制语义一致性，并结合LPIPS感知损失以在训练过程中提高视觉保真度。我们在两个基准数据集上验证了我们的方法：face2comics，将真实人脸翻译成漫画风格插图，以及edges2shoes，将边缘图翻译成逼真鞋类图像。实验结果表明，DiT结合CLIP条件和感知相似性目标实现了高质量、语义忠实的翻译，为配对图像到图像翻译任务提供了一个有前景的替代方案，替代基于GAN的模型。
英文摘要：
Image-to-image translation aims to learn a mapping between a source and a target domain, enabling tasks such as style transfer, appearance transformation, and domain adaptation. In this work, we explore a diffusion-based framework for image-to-image translation by adapting Diffusion Transformers (DiT), which combine the denoising capabilities of diffusion models with the global modeling power of transformers. To guide the translation process, we condition the model on image embeddings extracted from a pre-trained CLIP encoder, allowing for fine-grained and structurally consistent translations without relying on text or class labels. We incorporate both a CLIP similarity loss to enforce semantic consistency and an LPIPS perceptual loss to enhance visual fidelity during training. We validate our approach on two benchmark datasets: face2comics, which translates real human faces to comic-style illustrations, and edges2shoes, which translates edge maps to realistic shoe images. Experimental results demonstrate that DiT, combined with CLIP-based conditioning and perceptual similarity objectives, achieves high-quality, semantically faithful translations, offering a promising alternative to GAN-based models for paired image-to-image translation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
