基于双重溯因反事实推断的文本驱动图像编辑
Doubly Abductive Counterfactual Inference for Text-based Image Editing
AI总结：
针对单图像微调过拟合导致文本驱动图像编辑难以平衡可编辑性与保真度的问题，提出双重溯因反事实推断框架（DAC），通过UNet LoRA和文本编码器LoRA双重溯因，在定性和定量实验中均取得良好权衡，支持多种编辑意图。
摘要：
我们通过反事实推断研究单张图像的文本驱动图像编辑（TBIE），因为这是一种精确满足以下要求的优雅表述：编辑后的图像应保留原始图像的保真度。透过这一表述的视角，我们发现TBIE的关键在于现有技术难以在可编辑性和保真度之间取得良好的权衡，主要原因是单图像微调导致的过拟合。为此，我们提出了一种双重溯因反事实推断框架（DAC）。首先，我们将一个外生变量参数化为UNet LoRA，其溯因可以编码所有图像细节。其次，我们溯因另一个由文本编码器LoRA参数化的外生变量，该变量恢复了由第一次过拟合溯因所丢失的可编辑性。得益于第二次溯因，它专门编码了从编辑后到编辑前的视觉转换，其反演——减去LoRA——有效地将编辑前恢复为编辑后，从而完成编辑。通过大量实验，我们的DAC在可编辑性和保真度之间取得了良好的权衡。因此，我们可以支持广泛的用户编辑意图，包括添加、移除、操作、替换、风格迁移和面部改变，这些在定性和定量评估中都得到了广泛验证。代码在 https://github.com/xuesong39/DAC。
英文摘要：
We study text-based image editing (TBIE) of a single image by counterfactual inference because it is an elegant formulation to precisely address the requirement: the edited image should retain the fidelity of the original one. Through the lens of the formulation, we find that the crux of TBIE is that existing techniques hardly achieve a good trade-off between editability and fidelity, mainly due to the overfitting of the single-image fine-tuning. To this end, we propose a Doubly Abductive Counterfactual inference framework (DAC). We first parameterize an exogenous variable as a UNet LoRA, whose abduction can encode all the image details. Second, we abduct another exogenous variable parameterized by a text encoder LoRA, which recovers the lost editability caused by the overfitted first abduction. Thanks to the second abduction, which exclusively encodes the visual transition from post-edit to pre-edit, its inversion -- subtracting the LoRA -- effectively reverts pre-edit back to post-edit, thereby accomplishing the edit. Through extensive experiments, our DAC achieves a good trade-off between editability and fidelity. Thus, we can support a wide spectrum of user editing intents, including addition, removal, manipulation, replacement, style transfer, and facial change, which are extensively validated in both qualitative and quantitative evaluations. Codes are in https://github.com/xuesong39/DAC.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
