Text-to-image diffusion models can generate diverse, high-fidelity images based on user-provided text prompts. Recent research has extended these models to support text-guided image editing. While text guidance is an intuitive editing interface for users, it often fails to ensure the precise concept conveyed by users. To address this issue, we propose Custom-Edit, in which we (i) customize a diffusion model with a few reference images and then (ii) perform text-guided editing. Our key discovery is that customizing only language-relevant parameters with augmented prompts improves reference similarity significantly while maintaining source similarity. Moreover, we provide our recipe for each customization and editing process. We compare popular customization methods and validate our findings on two editing methods using various datasets.
VideoDreamer: Customized Multi-Subject Text-to-Video Generation with Disen-Mix Finetuning on Language-Video Foundation Models
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Beijing Film Academy（北京电影学院）；Beijing University of Technology（北京工业大学）
CustomTTT: Motion and Appearance Customized Video Generation via Test-Time Training
Chongqing University of Posts and Telecommunications（重庆邮电大学）；Great Bay University（大湾区大学）；Meituan（美团）；Jinan Inspur Data Technology Co., Ltd.（济南浪潮数据技术有限公司）
VideoMaker: Zero-shot Customized Video Generation with the Inherent Force of Video Diffusion Models
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；ARC Lab, Tencent PCG（腾讯平台与内容事业群ARC实验室）；Tencent AI Lab（腾讯AI实验室）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
DreamMix: Decoupling Object Attributes for Enhanced Editability in Customized Image Inpainting
Information and Communication Engineering（信息与通信工程）；Dalian University of Technology（大连理工大学）；ZMO.AI Inc.（ZMO.AI公司）；University of Electronic Science and Technology of China（电子科技大学）
Hierarchical Anti-Aesthetics: Protecting Facial Privacy against Customized Diffusion Models
School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）；China Mobile Information Technology Co., Ltd.（中国移动信息科技有限公司）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Center for Research on Intelligent Perception and Computing (CRIPAC), Institute of Automation Chinese Academy of Sciences (CASIA)（多模态人工智能系统国家重点实验室（MAIS）、智能感知与计算研究中心（CRIPAC）、中国科学院自动化研究所（CASIA））
StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaboration
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；United Arab Emirates University（阿拉伯联合酋长国大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
In-Context Brush: Zero-shot Customized Subject Insertion with Context-Aware Latent Space Manipulation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of Konstanz（康斯坦茨大学）；National Cheng-Kung University（国立成功大学）
Low-light image enhancement (LLIE) is a pervasive yet challenging problem, since: 1) low-light measurements may vary due to different imaging conditions in practice; 2) images can be enlightened subjectively according to diverse preferences by each individual. To tackle these two challenges, this paper presents a novel deep reinforcement learning based method, dubbed ReLLIE, for customized low-light enhancement. ReLLIE models LLIE as a markov decision process, i.e., estimating the pixel-wise image-specific curves sequentially and recurrently. Given the reward computed from a set of carefully c…
The ability to customize a trained Deep Neural Network (DNN) locally using user-specific data may greatly enhance user experiences, reduce development costs, and protect user's privacy. In this work, we propose to incorporate a novel Mixture of Experts (MOE) approach to accomplish this goal. This architecture comprises of a Global Expert (GE), a Local Expert (LE) and a Gating Network (GN). The GE is a trained DNN developed on a large training dataset representative of many potential users. After deployment on an embedded edge device, GE will be subject to customized, user-specific data (e.g., …
Despite excellent progress has been made, the performance of deep learning based algorithms still heavily rely on specific datasets, which are difficult to extend due to labor-intensive labeling. Moreover, because of the advancement of new applications, initial definition of data annotations might not always meet the requirements of new functionalities. Thus, there is always a great demand in customized data annotations. To address the above issues, we propose the Few-Shot Model Adaptation (FSMA) framework and demonstrate its potential on several important tasks on Faces. The FSMA first acquir…
University of Maryland, College Park（马里兰大学帕克分校）；Sun Yat-sen University（中山大学）；City University of Hong Kong（香港城市大学）
We introduce LUCSS, a language-based system for interactive col- orization of scene sketches, based on their semantic understanding. LUCSS is built upon deep neural networks trained via a large-scale repository of scene sketches and cartoon-style color images with text descriptions. It con- sists of three sequential modules. First, given a scene sketch, the segmenta- tion module automatically partitions an input sketch into individual object instances. Next, the captioning module generates the text description with spatial relationships based on the instance-level segmentation results. Fi- nal…
TV-3DG: Mastering Text-to-3D Customized Generation with Visual Prompt
Harbin Institute of Technology（哈尔滨工业大学）；Space AI, Li Auto（小鹏汽车太空人工智能）；School of Software, Tsinghua University（清华大学软件学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；Department of Automation, Tsinghua University（清华大学自动化系）
The best summary of a long video differs among different people due to its highly subjective nature. Even for the same person, the best summary may change with time or mood. In this paper, we introduce the task of generating customized video summaries through simple text. First, we train a deep architecture to effectively learn semantic embeddings of video frames by leveraging the abundance of image-caption data via a progressive and residual manner. Given a user-specific text description, our algorithm is able to select semantically relevant video segments and produce a temporally aligned vid…
We customize an end-to-end image compression framework for retina OCT images based on deep convolutional neural networks (CNNs). The customized compression scheme consists of three parts: data Preprocessing, compression CNNs, and reconstruction CNNs. Data preprocessing module reduces the speckle noise of the OCT images and the segments out the region of interest. We added customized skip connections between the compression CNNs and the reconstruction CNNs to reserve the detail information and trained the two nets together with the semantic segmented image patches from data preprocessing module…
Adv-CPG: A Customized Portrait Generation Framework with Facial Adversarial Attacks
School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University（西北工业大学人工智慧、光学与电子学院（iOPEN））；The University of Hong Kong（香港大学）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））
本文提出 Adv-CPG 框架，首次将面部对抗攻击引入定制肖像生成，通过轻量级局部 ID 加密器、加密增强器和多模态图像定制器实现隐私保护与细粒度个性化生成，实验表明其攻击成功率优于现有方法。
Improving Text-Instance Alignment Of Foreground Conditioned Out-Painting Via Customized Concept Embedding
College of Electronic and Information Engineering, Tongji University, Shanghai, China（电子信息工程学院，同济大学，上海，中国）；State Key Laboratory of Autonomous Intelligent Unmanned Systems, Frontiers Science Center for Intelligent Autonomous Systems, Ministry of Education, Shanghai, China（自主智能无人系统国家重点实验室，智能自主系统前沿科学中心，教育部，上海，中国）
University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Microsoft（微软）
Image-text contrastive learning models such as CLIP have demonstrated strong task transfer ability. The high generality and usability of these visual models is achieved via a web-scale data collection process to ensure broad concept coverage, followed by expensive pre-training to feed all the knowledge into model weights. Alternatively, we propose REACT, REtrieval-Augmented CusTomization, a framework to acquire the relevant web knowledge to build customized visual models for target domains. We retrieve the most relevant image-text pairs (~3% of CLIP pre-training data) from the web-scale databa…
Although LEGO sets have entertained generations of children and adults, the challenge of designing customized builds matching the complexity of real-world or imagined scenes remains too great for the average enthusiast. In order to make this feat possible, we implement a system that generates a LEGO brick model from 2D images. We design a novel solution to this problem that uses an octree-structured autoencoder trained on 3D voxelized models to obtain a feasible latent representation for model reconstruction, and a separate network trained to predict this latent representation from 2D images. …
Image description task has been invariably examined in a static manner with qualitative presumptions held to be universally applicable, regardless of the scope or target of the description. In practice, however, different viewers may pay attention to different aspects of the image, and yield different descriptions or interpretations under various contexts. Such diversity in perspectives is difficult to derive with conventional image description techniques. In this paper, we propose a customized image narrative generation task, in which the users are interactively engaged in the generation proc…
