We propose a novel algorithm, named Open-Edit, which is the first attempt on open-domain image manipulation with open-vocabulary instructions. It is a challenging task considering the large variation of image domains and the lack of training supervision. Our approach takes advantage of the unified visual-semantic embedding space pretrained on a general image-caption dataset, and manipulates the embedded visual features by applying text-guided vector arithmetic on the image feature maps. A structure-preserving image decoder then generates the manipulated images from the manipulated feature maps. We further propose an on-the-fly sample-specific optimization approach with cycle-consistency constraints to regularize the manipulated images and force them to preserve details of the source images. Our approach shows promising results in manipulating open-vocabulary color, texture, and high-level attributes for various scenarios of open-domain images.
East China Normal University（华东师范大学）；Xiaohongshu Inc.（小红书）；KLATASDS, MOE, China（中国教育部统计与数据科学重点实验室）；The Chinese University of Hong Kong（香港中文大学）；Zhejiang University（浙江大学）；Fudan University（复旦大学）；University of Oxford（牛津大学）；Shanghai Jiao Tong University（上海交通大学）；University of Science and Technology of China（中国科学技术大学）；Nanjing University（南京大学）
Beyond Editing Pairs: Fine-Grained Instructional Image Editing via Multi-Scale Learnable Regions
Department of Electronic Engineering and Computer Science University of California, Irvine（电子工程与计算机科学系 加州大学尔湾分校）；Department of Computer Science University of Alabama at Birmingham（计算机科学系 阿拉巴马大学伯明翰分校）
InstructP2P: Learning to Edit 3D Point Clouds with Text Instructions
ARC Lab, Tencent PCG（腾讯平台与内容事业群ARC实验室）；ShanghaiTech University（上海科技大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）；Shanghai Engineering Research Center of Energy Efficient and Custom AI IC（上海能效与定制人工智能集成电路工程技术研究中心）
SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models
University of Macau（澳门大学）；ARC Lab, Tencent PCG（腾讯平台与内容事业群ARC实验室）；Tencent AI Lab（腾讯人工智能实验室）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Tsinghua University（清华大学）
JAVEDIT: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curation
Zhejiang University（浙江大学）；Tencent Youtu Lab（腾讯优图实验室）；Nanjing University（南京大学）；University of Auckland（奥克兰大学）；Fudan University（复旦大学）；National University of Singapore（新加坡国立大学）
MIGE: Mutually Enhanced Multimodal Instruction-Based Image Generation and Editing
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）
Reasoning to Edit: Hypothetical Instruction-Based Image Editing with Visual Reasoning
Tencent Youtu Lab（腾讯云图实验室）；Sichuan University（四川大学）；University of the Chinese Academy of Sciences（中国科学院大学）；Fudan University（复旦大学）；Zhejiang University（浙江大学）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）
Beihang University（北京航空航天大学）；Xiaohongshu Inc（小红书公司）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Nanchang Institute of Technology（南昌工程学院）；Zhongguancun Laboratory（中关村实验室）
How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing
School of Artificial Intelligence, University of Chinese Academy of Science（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Language Technology Lab, University of Cambridge（剑桥大学语言技术实验室）；Peking University（北京大学）；South China University of Technology（华南理工大学）；Nanjing University（南京大学）
LocateEdit-Bench: A Benchmark for Instruction-Based Editing Localization
Institute of Automation, Chinese Academy of Sciences, Beijing, China（中国科学院自动化研究所）；Beijing Academy of Artificial Intelligence, Beijing, China（北京人工智能研究院）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学）；School of EEECS, Queen's University Belfast, Northern Ireland, UK（女王大学 Belfast 电子工程与计算机科学学院）；Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））；Peking University, Beijing, China（北京大学）
RefVideo-6M: A Reliable Reference-Based Dataset for Instructional Video Editing
Institute of Artificial Intelligence, China Telecom (TeleAI)（中国电信人工智能研究院（TeleAI））；The Chinese University of Hong Kong (CUHK)（香港中文大学）；Sun Yat-sen University（中山大学）；Fudan University（复旦大学）；Tsinghua University（清华大学）
