基于人类指令的动作驱动图像编辑
Action-based image editing guided by human instructions
AI总结：
本文提出一种动作敏感的文本图像编辑模型，通过对比学习识别动作差异，利用视频帧构建前后场景数据集，实现从静态起始图像生成动作最终场景的动态编辑。
摘要：
基于文本的图像编辑通常被视为一项静态任务，涉及根据人类指令对输入图像中的元素进行插入、删除或修改等操作。鉴于该任务的静态特性，本文旨在通过引入动作使其变为动态任务。通过这种方式，我们希望在保持物体视觉属性的同时，修改图像中物体的位置或姿态，以表现不同的动作。为实现这一具有挑战性的任务，我们提出了一种新模型，该模型通过学习识别对比性动作差异来对动作文本指令保持敏感。模型训练在新的数据集上进行，这些数据集通过从视频中提取帧来定义，展示动作发生前后的视觉场景。我们展示了使用基于动作的文本指令进行图像编辑的显著改进，以及较高的推理能力，使我们的模型能够将输入图像作为动作的起始场景，同时生成展示动作最终场景的新图像。
英文摘要：
Text-based image editing is typically approached as a static task that involves operations such as inserting, deleting, or modifying elements of an input image based on human instructions. Given the static nature of this task, in this paper, we aim to make this task dynamic by incorporating actions. By doing this, we intend to modify the positions or postures of objects in the image to depict different actions while maintaining the visual properties of the objects. To implement this challenging task, we propose a new model that is sensitive to action text instructions by learning to recognize contrastive action discrepancies. The model training is done on new datasets defined by extracting frames from videos that show the visual scenes before and after an action. We show substantial improvements in image editing using action-based text instructions and high reasoning capabilities that allow our model to use the input image as a starting scene for an action while generating a new image that shows the final scene of the action.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
