CLIPDrag：结合基于文本和基于拖拽的指令进行图像编辑
CLIPDrag: Combining Text-based and Drag-based Instructions for Image Editing
AI总结：
CLIPDrag首次结合文本与拖拽指令，通过全局-局部运动监督和快速点跟踪，实现扩散模型上的精确无歧义图像编辑，性能优于单一方法。
摘要：
精确且灵活的图像编辑仍然是计算机视觉中的一个基本挑战。根据修改区域的不同，大多数编辑方法可分为两种主要类型：全局编辑和局部编辑。在本文中，我们选择了两种最常见的编辑方法（即基于文本的编辑和基于拖拽的编辑）并分析了它们的缺点。具体而言，基于文本的方法往往无法精确描述所需的修改，而基于拖拽的方法则存在歧义问题。为解决这些问题，我们提出了CLIPDrag，一种新颖的图像编辑方法，它是首个在扩散模型上结合文本和拖拽信号以实现精确且无歧义操作的方法。为充分利用这两种信号，我们将文本信号视为全局指导，将拖拽点视为局部信息。然后，我们引入了一种新颖的全局-局部运动监督方法，通过适配预训练的语言-视觉模型（如CLIP）将文本信号整合到现有的基于拖拽的方法中。此外，我们还通过提出一种快速点跟踪方法来解决CLIPDrag中收敛缓慢的问题，该方法强制拖拽点向正确方向移动。大量实验表明，CLIPDrag优于现有的单一基于拖拽的方法或基于文本的方法。
英文摘要：
Precise and flexible image editing remains a fundamental challenge in computer vision. Based on the modified areas, most editing methods can be divided into two main types: global editing and local editing. In this paper, we choose the two most common editing approaches (ie text-based editing and drag-based editing) and analyze their drawbacks. Specifically, text-based methods often fail to describe the desired modifications precisely, while drag-based methods suffer from ambiguity. To address these issues, we proposed \textbf{CLIPDrag}, a novel image editing method that is the first to combine text and drag signals for precise and ambiguity-free manipulations on diffusion models. To fully leverage these two signals, we treat text signals as global guidance and drag points as local information. Then we introduce a novel global-local motion supervision method to integrate text signals into existing drag-based methods by adapting a pre-trained language-vision model like CLIP. Furthermore, we also address the problem of slow convergence in CLIPDrag by presenting a fast point-tracking method that enforces drag points moving toward correct directions. Extensive experiments demonstrate that CLIPDrag outperforms existing single drag-based methods or text-based methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
