在扩散模型中将文本对齐到图像比你想象的更容易
Aligning Text to Image in Diffusion Models is Easier Than You Think
AI总结：
本文提出SoftREPA方法，通过对比学习提升文本与图像表示的对齐效果，减少计算开销，增强语义一致性。
摘要：
尽管生成建模的最新进展显著提高了文本-图像对齐能力，但文本和图像表示之间仍存在一些残余不一致。一些方法通过偏好优化等方法对模型进行微调，但需要定制化的数据集。与这些方法不同，我们从表示对齐的角度重新审视这一挑战，这种方法随着REPA（表示对齐）的成功而变得流行。我们首先认为，传统的文本到图像（T2I）扩散模型通常通过最小化分数匹配或流匹配损失来训练，使用配对的图像和文本数据（即正样本对），从表示对齐的角度来看是次优的。相反，通过对比学习利用现有数据集作为正样本对和负样本对，可以实现更好的对齐。为了在预训练模型上实现高效的对齐，我们提出了SoftREPA——一种轻量级的对比微调策略，利用软文本标记进行表示对齐。这种方法通过添加少于100万可训练参数到预训练模型中，以最小的计算开销提高了对齐效果。我们的理论分析表明，我们的方法明确增加了文本和图像表示之间的互信息，从而增强了语义一致性。在文本到图像生成和文本引导的图像编辑任务中的实验结果验证了我们的方法在提高T2I生成模型语义一致性方面的有效性。
英文摘要：
While recent advancements in generative modeling have significantly improved text-image alignment, some residual misalignment between text and image representations still remains. Some approaches address this issue by fine-tuning models in terms of preference optimization, etc., which require tailored datasets. Orthogonal to these methods, we revisit the challenge from the perspective of representation alignment-an approach that has gained popularity with the success of REPresentation Alignment (REPA). We first argue that conventional text-to-image (T2I) diffusion models, typically trained on paired image and text data (i.e., positive pairs) by minimizing score matching or flow matching losses, is suboptimal from the standpoint of representation alignment. Instead, a better alignment can be achieved through contrastive learning that leverages existing dataset as both positive and negative pairs. To enable efficient alignment with pretrained models, we propose SoftREPA- a lightweight contrastive fine-tuning strategy that leverages soft text tokens for representation alignment. This approach improves alignment with minimal computational overhead by adding fewer than 1M trainable parameters to the pretrained model. Our theoretical analysis demonstrates that our method explicitly increases the mutual information between text and image representations, leading to enhanced semantic consistency. Experimental results across text-to-image generation and text-guided image editing tasks validate the effectiveness of our approach in improving the semantic consistency of T2I generative models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
