Natural language instructions are a powerful interface for editing the outputs of text-to-image diffusion models. However, several challenges need to be addressed: 1) underspecification (the need to model the implicit meaning of instructions) 2) grounding (the need to localize where the edit has to be performed), 3) faithfulness (the need to preserve the elements of the image not affected by the edit instruction). Current approaches focusing on image editing with natural language instructions rely on automatically generated paired data, which, as shown in our investigation, is noisy and sometimes nonsensical, exacerbating the above issues. Building on recent advances in segmentation, Chain-of-Thought prompting, and visual question answering, we significantly improve the quality of the paired data. In addition, we enhance the supervision signal by highlighting parts of the image that need to be changed by the instruction. The model fine-tuned on the improved data is capable of performing fine-grained object-centric edits better than state-of-the-art baselines, mitigating the problems outlined above, as shown by automatic and human evaluations. Moreover, our model is capable of generalizing to domains unseen during training, such as visual metaphors.
EPEdit: Redefining Image Editing with Generative AI and User-Centric Design
University of Science, VNU-HCM(越南胡志明市科学大学);Vietnam National University, Ho Chi Minh City(越南国家大学,胡志明市);University of Information Technology, VNU-HCM(越南胡志明市信息科技大学);University of Dayton(Dayton 大学)
EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing
The University of Texas at Austin(德克萨斯大学奥斯汀分校);University of California, Los Angeles(加州大学洛杉矶分校);Microsoft AI Superintelligence(微软人工智能超智实验室);Lambda, Inc(Lambda公司)
Object-Aware Background-Controlled Editing via Weighted Velocity Guidance
Carnegie Mellon University(卡内基梅隆大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)
Uncovering Entity Identity Confusion in Multimodal Knowledge Editing
New Laboratory of Pattern Recognition (NLPR)(模式识别新实验室);State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS)(多模态人工智能系统国家重点实验室);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Beijing University of Posts and Telecommunications(北京邮电大学);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);Huazhong University of Science and Technology(华中科技大学);Shandong University(山东大学)
S$^2$Edit: Text-Guided Image Editing with Precise Semantic and Spatial Control
Department of Computer Science, University of Toronto(多伦多大学计算机科学系);ModiFace Inc.(ModiFace公司);Department of Computer Science, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校计算机科学系)
MiVE: Multiscale Vision-language features for reference-guided video Editing
MT Lab, Meitu Inc., Beijing 100083, China(美图实验室,美图公司,北京100083,中国);Department of Computer Science and Technology, BNRist, IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China(计算机科学与技术系,BNRist,IDG/麦戈文脑研究学院,清华大学,北京100084,中国);Beijing University of Posts(北京邮电大学)
Southeast University(东南大学);Max Planck Institute for Informatics(马克斯·普朗克信息研究所);Shanghai Jiao Tong University(上海交通大学);StepFun;University of California, Berkeley(加州大学伯克利分校);University of California, Merced(加州大学默塞德分校)
JAVEDIT: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curation
Zhejiang University(浙江大学);Tencent Youtu Lab(腾讯优图实验室);Nanjing University(南京大学);University of Auckland(奥克兰大学);Fudan University(复旦大学);National University of Singapore(新加坡国立大学)
SplitFlow: Flow Decomposition for Inversion-Free Text-to-Image Editing
Harvard AI and Robotics Lab, Harvard University(哈佛人工智能与机器人实验室,哈佛大学);École des Ponts, Institut Polytechnique de Paris(巴黎高等工业学院);New York University Abu Dhabi(纽约大学阿布扎克分校);Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University(哈佛大学自然与人工智能研究学院)
EditTransfer++: Toward Faithful and Efficient Visual-Prompt-Guided Image Editing
School of Information and Communication Engineering and the State Key Laboratory of Media Convergence and Communication, Communication University of China(信息与通信工程学院和媒体融合与通信国家重点实验室,中国传媒大学);ShowLab, National University of Singapore(新加坡国立大学ShowLab)