Text as Neural Operator: Image Manipulation by Text Instruction
Google Research（谷歌研究院）；University of California, Merced（加利福尼亚大学默塞德分校）；Carnegie Mellon University（卡内基梅隆大学）；Waymo；University of Michigan（密歇根大学）；Georgia Institute of Technology（佐治亚理工学院）
Tianhao Zhang, Hung-Yu Tseng, Lu Jiang, Weilong Yang, Honglak Lee, Irfan Essa
In recent years, text-guided image manipulation has gained increasing attention in the multimedia and computer vision community. The input to conditional image generation has evolved from image-only to multimodality. In this paper, we study a setting that allows users to edit an image with multiple objects using complex text instructions to add, remove, or change the objects. The inputs of the task are multimodal including (1) a reference image and (2) an instruction in natural language that describes desired modifications to the image. We propose a GAN-based method to tackle this problem. The key idea is to treat text as neural operators to locally modify the image feature. We show that the proposed model performs favorably against recent strong baselines on three public datasets. Specifically, it generates images of greater fidelity and semantic relevance, and when used as a image query, leads to better retrieval performance.
Nara Institute of Science Technology（奈良先端科学技术大学院大学）；RIKEN, Center for Advanced Intelligence Project AIP（理化学研究所先进智能项目中心）；PRESTO, Japan Science and Technology Agency（日本科学技术振兴机构先驱研究）；RIKEN（理化学研究所）；Japan Science and Technology Agency（日本科学技术振兴机构）
We propose an interactive image-manipulation system with natural language instruction, which can generate a target image from a source image and an instruction that describes the difference between the source and the target image. The system makes it possible to modify a generated image interactively and make natural language conditioned image generation more controllable. We construct a neural network that handles image vectors in latent space to transform the source vector to the target vector by using the vector of instruction. The experimental results indicate that the proposed framework s…
Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）
Conditional image generation is an active research topic including text2image and image translation. Recently image manipulation with linguistic instruction brings new challenges of multimodal conditional generation. However, traditional conditional image generation models mainly focus on generating high-quality and visually realistic images, and lack resolving the partial consistency between image and instruction. To address this issue, we propose an Increment Reasoning Generative Adversarial Network (IR-GAN), which aims to reason the consistency between visual increment in images and semanti…
SIGMA: Semantic-Difference Instruction-Grounding Mask Annotator for Text-Driven Image Manipulation Localization
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Guangdong Provincial Key Laboratory of Intelligent Information Processing and Shenzhen Key Laboratory of Media Security（广东省智能信息处理重点实验室和深圳媒体安全重点实验室）；Shenzhen University of Advanced Technology and Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（深圳先进技术大学和深圳先进技术研究所，中国科学院）；Alibaba Group（阿里巴巴集团）；Shenzhen MSU-BIT University（深圳MSU-BIT大学）
East China Normal University（华东师范大学）；Xiaohongshu Inc.（小红书）；KLATASDS, MOE, China（中国教育部统计与数据科学重点实验室）；The Chinese University of Hong Kong（香港中文大学）；Zhejiang University（浙江大学）；Fudan University（复旦大学）；University of Oxford（牛津大学）；Shanghai Jiao Tong University（上海交通大学）；University of Science and Technology of China（中国科学技术大学）；Nanjing University（南京大学）
LIVE: Leveraging Image Manipulation Priors for Instruction-based Video Editing
Nankai University（南开大学）；Pengcheng Laboratory（鹏城实验室）；Kuaishou Technology（快手科技）；Nankai International Advanced Research Institute (SHENZHEN·FUTIAN)（南开国际先进研究院（深圳·福田））
Don't Let Your Robot be Harmful: Responsible Robotic Manipulation via Safety-as-Policy
Hong Kong Polytechnic University（香港理工大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Hamburg（汉堡大学）；Peng Cheng Laboratory（鹏城实验室）；Agile Robots AG（智元机器人股份有限公司）
The Chinese University of Hong Kong（香港中文大学）；Adobe Research（Adobe研究院）
We propose a novel algorithm, named Open-Edit, which is the first attempt on open-domain image manipulation with open-vocabulary instructions. It is a challenging task considering the large variation of image domains and the lack of training supervision. Our approach takes advantage of the unified visual-semantic embedding space pretrained on a general image-caption dataset, and manipulates the embedded visual features by applying text-guided vector arithmetic on the image feature maps. A structure-preserving image decoder then generates the manipulated images from the manipulated feature maps…
Inria（法国国家信息与自动化研究所）；École normale supérieure（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）；IIIT Hyderabad（海得拉巴国际信息技术学院）
In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions. Yet, robotic manipulation is extremely challenging as it requires fine-grained motor control, long-term memory as well as generalization to previously unseen tasks and environments. To address these challenges, we propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations while (iii) keeping track of the full h…
Object-Centric Instruction Augmentation for Robotic Manipulation
School of Computer Science, East China Normal University（华东师范大学计算机科学学院）；Midea Group（美的集团）；Department of Mathematics, School of Science, Shanghai University（上海大学理学院数学系）
Text-guided image manipulation tasks have recently gained attention in the vision-and-language community. While most of the prior studies focused on single-turn manipulation, our goal in this paper is to address the more challenging multi-turn image manipulation (MTIM) task. Previous models for this task successfully generate images iteratively, given a sequence of instructions and a previously generated image. However, this approach suffers from under-generation and a lack of generated quality of the objects that are described in the instructions, which consequently degrades the overall perfo…
Recently, text-guided image manipulation has received increasing attention in the research field of multimedia processing and computer vision due to its high flexibility and controllability. Its goal is to semantically manipulate parts of an input reference image according to the text descriptions. However, most of the existing works have the following problems: (1) text-irrelevant content cannot always be maintained but randomly changed, (2) the performance of image manipulation still needs to be further improved, (3) only can manipulate descriptive attributes. To solve these problems, we pro…
LocateEdit-Bench: A Benchmark for Instruction-Based Editing Localization
Institute of Automation, Chinese Academy of Sciences, Beijing, China（中国科学院自动化研究所）；Beijing Academy of Artificial Intelligence, Beijing, China（北京人工智能研究院）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学）；School of EEECS, Queen's University Belfast, Northern Ireland, UK（女王大学 Belfast 电子工程与计算机科学学院）；Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））；Peking University, Beijing, China（北京大学）
InstructP2P: Learning to Edit 3D Point Clouds with Text Instructions
ARC Lab, Tencent PCG（腾讯平台与内容事业群ARC实验室）；ShanghaiTech University（上海科技大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）；Shanghai Engineering Research Center of Energy Efficient and Custom AI IC（上海能效与定制人工智能集成电路工程技术研究中心）
National Taiwan University（国立台湾大学）；UCLA（加州大学洛杉矶分校）；Purdue University（普渡大学）；NVIDIA（英伟达）
We tackle the problem of target-free text-guided image manipulation, which requires one to modify the input reference image based on the given text instruction, while no ground truth target image is observed during training. To address this challenging task, we propose a Cyclic-Manipulation GAN (cManiGAN) in this paper, which is able to realize where and how to edit the image regions of interest. Specifically, the image editor in cManiGAN learns to identify and complete the input image, while cross-modal interpreter and reasoner are deployed to verify the semantic correctness of the output ima…
University of California, Santa Cruz（加州大学圣克鲁兹分校）；University of Michigan, Ann Arbor（密歇根大学安娜堡分校）
Benefiting from language flexibility and compositionality, humans naturally intend to use language to command an embodied agent for complex tasks such as navigation and object manipulation. In this work, we aim to fill the blank of the last mile of embodied agents -- object manipulation by following human guidance, e.g., "move the red mug next to the box while keeping it upright." To this end, we introduce an Automatic Manipulation Solver (AMSolver) system and build a Vision-and-Language Manipulation benchmark (VLMbench) based on it, containing various language instructions on categorized robo…
Compressor-VLA: Instruction-Guided Visual Token Compression for Efficient Robotic Manipulation
School of Information Science and Technology, Beijing University of Technology（信息科学与技术学院，北京理工大学）；LiAuto Inc.（LiAuto公司）；Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing University of Technology（计算智能与智能系统北京市重点实验室，北京理工大学）
