arXiv 2407.186562024-07-26cs.CV

Auto DragGAN：以自回归方式编辑生成式图像流形

Auto DragGAN: Editing the Generative Image Manifold in an Autoregressive Manner

Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Wuhan AI Research（武汉人工智能研究院）

Pengxiang Cai, Zhiwei Liu, Guibo Zhu, Yunfang Niu, Jinqiao Wang

备注：: This paper has been accepted as a poster paper for ACM Multimedia 2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 Auto DragGAN，首次采用基于回归的网络学习 StyleGAN 潜在码在拖拽过程中的变化模式，通过自回归方式分解移动过程，在像素级粒度上实现了最先进的推理速度与图像编辑性能。

摘要：

像素级细粒度图像编辑仍是一个开放性挑战。先前的工作未能在控制粒度与推理速度之间实现理想权衡。它们要么无法实现像素级细粒度控制，要么推理速度需要优化。为解决此问题，本文首次采用基于回归的网络学习 StyleGAN 潜在码在图像拖拽过程中的变化模式。该方法以极小的时间成本实现拖拽编辑中的像素级精度。用户可在任意 GAN 生成的图像上指定控制点及其对应目标点，我们的方法会将每个控制点移动至其对应目标点。通过实验分析，我们发现控制点至目标点的移动距离较短时会产生高保真编辑图像，因为模型仅需预测少部分像素的移动。为此，我们将整个移动过程分解为多个子过程。具体而言，我们开发了一个基于 transformer 编码器-解码器的网络 Latent Predictor，以自回归方式预测从控制点到目标点的潜在码运动轨迹。此外，为增强预测稳定性，我们引入名为 Latent Regularizer 的组件，旨在将潜在码运动约束在自然图像分布内。大量实验表明，我们的方法在像素级粒度上实现了最先进的推理速度与图像编辑性能。

英文摘要：

Pixel-level fine-grained image editing remains an open challenge. Previous works fail to achieve an ideal trade-off between control granularity and inference speed. They either fail to achieve pixel-level fine-grained control, or their inference speed requires optimization. To address this, this paper for the first time employs a regression-based network to learn the variation patterns of StyleGAN latent codes during the image dragging process. This method enables pixel-level precision in dragging editing with little time cost. Users can specify handle points and their corresponding target points on any GAN-generated images, and our method will move each handle point to its corresponding target point. Through experimental analysis, we discover that a short movement distance from handle points to target points yields a high-fidelity edited image, as the model only needs to predict the movement of a small portion of pixels. To achieve this, we decompose the entire movement process into multiple sub-processes. Specifically, we develop a transformer encoder-decoder based network named 'Latent Predictor' to predict the latent code motion trajectories from handle points to target points in an autoregressive manner. Moreover, to enhance the prediction stability, we introduce a component named 'Latent Regularizer', aimed at constraining the latent code motion within the distribution of natural images. Extensive experiments demonstrate that our method achieves state-of-the-art (SOTA) inference speed and image editing performance at the pixel-level granularity.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。