Recent remarkable improvements in large-scale text-to-image generative models have shown promising results in generating high-fidelity images. To further enhance editability and enable fine-grained generation, we introduce a multi-input-conditioned image composition model that incorporates a sketch as a novel modal, alongside a reference image. Thanks to the edge-level controllability using sketches, our method enables a user to edit or complete an image sub-part with a desired structure (i.e., sketch) and content (i.e., reference image). Our framework fine-tunes a pre-trained diffusion model to complete missing regions using the reference image while maintaining sketch guidance. Albeit simple, this leads to wide opportunities to fulfill user needs for obtaining the in-demand images. Through extensive experiments, we demonstrate that our proposed method offers unique use cases for image manipulation, enabling user-driven modifications of arbitrary scenes.
Towards Realistic Face Photo-Sketch Synthesis via Composition-Aided GANs
Hangzhou Dianzi University(杭州电子科技大学);Xidian University(西安电子科技大学);Hefei University of Technology(合肥工业大学);The University of Sydney(悉尼大学);University of Chinese Academy of Sciences(中国科学院大学)
School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学学院);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳))
SketchAnimator: Animate Sketch via Motion Customization of Text-to-Video Diffusion Models
PRIS, School of Artificial Intelligence, Beijing University of Posts(PRIS人工智能学院,北京邮电大学);SketchX, CVSSP, University of Surrey(SketchX,计算机视觉与模式识别研究所,萨里大学)
U-Sketch: An Efficient Approach for Sketch to Image Diffusion Models
Artificial Intelligence and Learning Systems Laboratory(人工智能与学习系统实验室);School of Electrical and Computer Engineering(电气与计算机工程学院);National Technical University of Athens(雅典国立技术大学)
Center for Research on Intelligent Perception and Computing, NLPR, CASIA(中国科学院自动化研究所模式识别国家重点实验室智能感知与计算研究中心);University of Chinese Academy of Sciences(中国科学院大学);Meituandianping Group(美团点评集团);Artificial Intelligence Research, CAS(中国科学院人工智能研究院);Center for Excellence in Brain Science and Intelligence Technology, CAS(中国科学院脑科学与智能技术卓越创新中心)
近年来,人脸肖像编辑技术取得了很大的进步。然而,先前的方法或者1)对预定义的面部属性进行操作,缺乏控制高级语义面部组件的形状的灵活性(例如,眼睛、鼻子、嘴巴),或者2)将手动编辑的遮罩或草图作为可观察变化的中间表示,但是这样的附加输入通常需要额外的努力来获得。为了打破现有方法的局限性(例如形状、遮罩或草图),我们提出了一种新的框架r-FACE(Reference-guided FACE Component Editing),用于进行具有几何变化的多样化和可控的面组件编辑。具体而言,r-FACE以图像修补模型为主干,利用基准图像作为控制面部成分的形状的条件。为了鼓励框架集中在目标人脸组件,一个例子引导的注意力模块被设计为融合注意力特征和目标人脸组件特征提取的参考图像。通过大量的实验验证和比较,我们验证了所提出的框架的有效性。
Sketch2Manga: Shaded Manga Screening from Sketch with Diffusion Models
School of Computing and Information Sciences, Saint Francis University(圣弗朗西斯大学计算与信息科学学院);Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程学系)
Weave Forcing: Compositional Memory Routing for Interactive Long Video Generation
University of Electronic Science and Technology of China (UESTC)(电子科技大学);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)(穆罕默德·本·扎耶德人工智能大学)
Sketch2Human: Deep Human Generation with Disentangled Geometry and Appearance Control
School of Creative Media, City University of Hong Kong(香港城市大学创意媒体学院);Department of Computer Science & Engineering, HKUST(香港科技大学计算机科学与工程学系);Shenzhen Research Institute of Big Data, Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)深圳大数据研究院)
CogCanvas: A Benchmark for Evaluating Multi-Subject Reference-Based Image Generation
University of Science, Ho Chi Minh City, Vietnam(胡志明市理科大学);University of Dayton, Ohio, United States(代顿大学);Vietnam National University, Ho Chi Minh City, Vietnam(越南国家大学胡志明市分校)
UniRef-Image-Edit: Towards Scalable and Consistent Multi-Reference Image Editing
Tsinghua University(清华大学);Kuaishou Technology(快手科技);Hong Kong Polytechnic University(香港理工大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));CUHK MMLab(香港中文大学MMLab)
MixSA: Training-free Reference-based Sketch Extraction via Mixture-of-Self-Attention
School of Computer Sciences, Shaanxi Normal University(陕西师范大学计算机科学学院);School of Computing and Information Systems, Singapore Management University(新加坡管理大学计算与信息系统学院)