Image editing involves a variety of complex tasks and requires efficient and precise manipulation techniques. In this paper, we present MagicQuill, an integrated image editing system that enables swift actualization of creative ideas. Our system features a streamlined yet functionally robust interface, allowing for the articulation of editing operations (e.g., inserting elements, erasing objects, altering color) with minimal input. These interactions are monitored by a multimodal large language model (MLLM) to anticipate editing intentions in real time, bypassing the need for explicit prompt entry. Finally, we apply a powerful diffusion prior, enhanced by a carefully learned two-branch plug-in module, to process editing requests with precise control. Experimental results demonstrate the effectiveness of MagicQuill in achieving high-quality image edits. Please visit https://magic-quill.github.io to try out our system.
Generative adversarial networks (GANs) have enabled photorealistic image synthesis and editing. However, due to the high computational cost of large-scale generators (e.g., StyleGAN2), it usually takes seconds to see the results of a single edit on edge devices, prohibiting interactive user experience. In this paper, we take inspirations from modern rendering software and propose Anycost GAN for interactive natural image editing. We train the Anycost GAN to support elastic resolutions and channels for faster image generation at versatile speeds. Running subsets of the full generator produce ou…
Tsinghua University（清华大学）；Tencent AI Lab（腾讯人工智能实验室）；ByteDance Inc.（字节跳动公司）
Existing 3D-aware facial generation methods face a dilemma in quality versus editability: they either generate editable results in low resolution or high-quality ones with no editing flexibility. In this work, we propose a new approach that brings the best of both worlds together. Our system consists of three major components: (1) a 3D-semantics-aware generative model that produces view-consistent, disentangled face images and semantic masks; (2) a hybrid GAN inversion approach that initialize the latent codes from the semantic and texture encoder, and further optimized them for faithful recon…
SceneCraft: Interactive System for Image Editing via Scene Graph
University of Science, Ho Chi Minh, Vietnam（胡志明市理科大学）；Vietnam National University, Ho Chi Minh, Vietnam（越南国家大学胡志明市）；University of Dayton, Dayton, Ohio, USA（代顿大学）
Pay Attention and Move Better: Harnessing Attention for Interactive Motion Generation and Training-free Editing
Tsinghua University（清华大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；The University of Hong Kong（香港大学）；Shanghai AI Laboratory（上海人工智能实验室）
I2E: From Image Pixels to Actionable Interactive Environments for Text-Guided Image Editing
Huazhong University of Science and Technology（华中科技大学）；Kuaishou Technology（快手科技）；Central China Normal University（华中师范大学）；Tsinghua University（清华大学）；Shanghai AI Laboratory（上海人工智能实验室）
FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Science（中国科学院大学人工智能学院）；Peng Cheng Laboratory, Shenzhen, China（鹏城实验室）；Wuhan AI Research, Wuhan, China（武汉人工智能研究所）
ArtisanGS: Interactive Tools for Gaussian Splat Selection with AI and Human in the Loop
NVIDIA；NVIDIA Canada（NVIDIA 加拿大）；University of Toronto Canada（多伦多大学 加拿大）；NVIDIA France（NVIDIA 法国）；University of Toronto（多伦多大学）；Institute for Clarity in Documentation Dublin Ohio USA（文档清晰研究所 俄亥俄州 大致）；Inria Paris-Rocquencourt（法国巴黎-罗克琴特研究所）；Rajiv Gandhi University Doimukh Arunachal Pradesh India（拉吉夫·甘地大学 亚当穆克 阿鲁纳恰尔邦 印度）；Tsinghua University Haidian Qu Beijing Shi China（清华大学 海淀区 北京市 中国）；Palmer Research Laboratories San Antonio Texas USA（帕勒研究中心 肯塔基州 威斯康星州 美国）
University of Science and Technology of China（中国科学技术大学）；Snap Inc（斯奈普公司）；Microsoft Cloud AI（微软云人工智能）；City University of Hong Kong（香港城市大学）
Despite the recent success of face image generation with GANs, conditional hair editing remains challenging due to the under-explored complexity of its geometry and appearance. In this paper, we present MichiGAN (Multi-Input-Conditioned Hair Image GAN), a novel conditional image generation method for interactive portrait hair manipulation. To provide user control over every major hair visual factor, we explicitly disentangle hair into four orthogonal attributes, including shape, structure, appearance, and background. For each of them, we design a corresponding condition module to represent, pr…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Significant progress has been made by the advances in Generative Adversarial Networks (GANs) for image generation. However, there lacks enough understanding of how a realistic image is generated by the deep representations of GANs from a random vector. This chapter gives a summary of recent works on interpreting deep generative models. The methods are categorized into the supervised, the unsupervised, and the embedding-guided approaches. We will see how the human-understandable concepts that emerge in the learned representation can be identified and used for interactive image generation and ed…
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shenyang Institute of Computing Technology, Chinese Academy of Sciences（中国科学院沈阳计算技术研究所）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Microsoft Dynamics 365 AI Research（微软 Dynamics 365 人工智能研究院）；Duke University（杜克大学）；Microsoft Research（微软研究院）
Most existing text-to-image synthesis tasks are static single-turn generation, based on pre-defined textual descriptions of images. To explore more practical and interactive real-life applications, we introduce a new task - Interactive Image Editing, where users can guide an agent to edit images via multi-turn textual commands on-the-fly. In each session, the agent takes a natural language description from the user as the input and modifies the image generated in the previous turn to a new design, following the user description. The main challenges in this sequential and interactive image gene…
Trinity College Dublin（都柏林圣三一学院）；Adobe Research（Adobe研究）
Interactive object cutout tools are the cornerstone of the image editing workflow. Recent deep-learning based interactive segmentation algorithms have made significant progress in handling complex images and rough binary selections can typically be obtained with just a few clicks. Yet, deep learning techniques tend to plateau once this rough selection has been reached. In this work, we interpret this plateau as the inability of current algorithms to sufficiently leverage each user interaction and also as the limitations of current training/testing datasets. We propose a novel interactive archi…
Neural style transfer (NST) can create impressive artworks by transferring reference style to content image. Current image-to-image NST methods are short of fine-grained controls, which are often demanded by artistic editing. To mitigate this limitation, we propose a drawing-like interactive style transfer (IST) method, by which users can interactively create a harmonious-style image. Our IST method can serve as a brush, dip style from anywhere, and then paint to any region of the target content image. To determine the action scope, we formulate a fluid simulation algorithm, which takes styles…
IIIT, Hyderabad（海得拉巴国际信息技术研究所）；University of Bath（巴斯大学）
This paper proposes a video editor based on OpenShot with several state-of-the-art facial video editing algorithms as added functionalities. Our editor provides an easy-to-use interface to apply modern lip-syncing algorithms interactively. Apart from lip-syncing, the editor also uses audio and facial re-enactment to generate expressive talking faces. The manual control improves the overall experience of video editing without missing out on the benefits of modern synthetic video generation algorithms. This control enables us to lip-sync complex dubbed movie scenes, interviews, television shows,…
The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
Facial editing is an important task in vision and graphics with numerous applications. However, existing works are incapable to deliver a continuous and fine-grained editing mode (e.g., editing a slightly smiling face to a big laughing one) with natural interactions with users. In this work, we propose Talk-to-Edit, an interactive facial editing framework that performs fine-grained attribute manipulation through dialog between the user and the system. Our key insight is to model a continual "semantic field" in the GAN latent space. 1) Unlike previous works that regard the editing as traversing…
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Texas A&M University（德克萨斯农工大学）
Sketch-based image editing aims to synthesize and modify photos based on the structural information provided by the human-drawn sketches. Since sketches are difficult to collect, previous methods mainly use edge maps instead of sketches to train models (referred to as edge-based models). However, sketches display great structural discrepancy with edge maps, thus failing edge-based models. Moreover, sketches often demonstrate huge variety among different users, demanding even higher generalizability and robustness for the editing model to work. In this paper, we propose Deep Plastic Surgery, a …
Adobe Research（奥多比研究院）；Australian National University（澳大利亚国立大学）
Controllable image synthesis with user scribbles is a topic of keen interest in the computer vision community. In this paper, for the first time we study the problem of photorealistic image synthesis from incomplete and primitive human paintings. In particular, we propose a novel approach paint2pix, which learns to predict (and adapt) "what a user wants to draw" from rudimentary brushstroke inputs, by learning a mapping from the manifold of incomplete human paintings to their realistic renderings. When used in conjunction with recent works in autonomous painting agents, we show that paint2pix …
