arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新，休息一下吧，周末愉快~~
arXiv 2406.06258cs.CV
收藏

通过视图迭代自注意力控制实现免调优视觉定制

Tuning-Free Visual Customization via View Iterative Self-Attention Control

Tsinghua University（清华大学）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Nanchang University（南昌大学）；Jiluan Academy, Nanchang University（南昌大学吉鸾学院）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）

Xiaojie Li, Chenghao Gu, Shuzhao Xie, Yunpeng Bai, Weixiang Zhang, Zhi Wang

备注：
Under review
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对 LoRA 等方法依赖训练和多张参考图的问题，本文提出免训练的 VisCtrl，通过 DDIM 反演与迭代自注意力特征注入，实现单参考图的图像、视频及 3D 场景个性化编辑。

摘要：

微调扩散模型支持在多种视觉模态上开展广泛的个性化生成与编辑应用。尽管低秩适配（LoRA）加快了微调过程，但它仍需要多张参考图像和耗时的训练，这限制了其在大规模与实时应用中的可扩展性。本文提出 View Iterative Self-Attention Control（VisCtrl，视图迭代自注意力控制）来应对这一挑战。具体而言，VisCtrl 是一种无需训练的方法，它将用户指定主体的外观和结构注入目标图像中的另一主体，而不同于以往需要微调模型的方法。首先，我们通过 DDIM 反演获得参考图像和目标图像的初始噪声。随后，在去噪阶段，参考图像的特征通过自注意力机制被注入目标图像。值得注意的是，通过迭代执行这一特征注入过程，我们确保参考图像特征逐渐融合到目标图像中。该方法仅需一张参考图像并在少数几个去噪步骤内即可产生一致且协调的编辑结果。此外，得益于即插即用的架构设计以及为多视图编辑提出的 Feature Gradual Sampling（特征渐进采样）策略，我们的方法能够轻松扩展到复杂视觉领域的编辑。大量实验表明，VisCtrl 在一系列任务上均有效，包括图像、视频和三维场景的个性化编辑。

英文摘要：

Fine-Tuning Diffusion Models enable a wide range of personalized generation and editing applications on diverse visual modalities. While Low-Rank Adaptation (LoRA) accelerates the fine-tuning process, it still requires multiple reference images and time-consuming training, which constrains its scalability for large-scale and real-time applications. In this paper, we propose \textit{View Iterative Self-Attention Control (VisCtrl)} to tackle this challenge. Specifically, VisCtrl is a training-free method that injects the appearance and structure of a user-specified subject into another subject in the target image, unlike previous approaches that require fine-tuning the model. Initially, we obtain the initial noise for both the reference and target images through DDIM inversion. Then, during the denoising phase, features from the reference image are injected into the target image via the self-attention mechanism. Notably, by iteratively performing this feature injection process, we ensure that the reference image features are gradually integrated into the target image. This approach results in consistent and harmonious editing with only one reference image in a few denoising steps. Moreover, benefiting from our plug-and-play architecture design and the proposed Feature Gradual Sampling strategy for multi-view editing, our method can be easily extended to edit in complex visual domains. Extensive experiments show the efficacy of VisCtrl across a spectrum of tasks, including personalized editing of images, videos, and 3D scenes.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑