arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.19145cs.CLcs.CV
收藏

学习忠实地遵循以对象为中心的图像编辑指令

Learning to Follow Object-Centric Image Editing Instructions Faithfully

Columbia University(哥伦比亚大学)

Tuhin Chakrabarty, Kanishk Singh, Arkadiy Saakyan, Smaranda Muresan

AI总结:

本文针对自然语言图像编辑中的欠明确性、接地和忠实性问题,通过改进自动配对数据质量并突出需修改区域来增强监督,微调后的模型在细粒度对象编辑上优于现有基线并具备跨领域泛化能力。

摘要:

自然语言指令是编辑文本到图像扩散模型输出的一种强大接口。然而,仍需要解决几个挑战:1)欠明确性(需要建模指令的隐含含义);2)接地(需要定位必须执行编辑的位置);3)忠实性(需要保留图像中不受编辑指令影响的元素)。当前专注于使用自然语言指令进行图像编辑的方法依赖于自动生成的配对数据,而正如我们的调查所示,这些数据存在噪声,有时甚至毫无意义,从而加剧了上述问题。基于分割、Chain-of-Thought提示和视觉问答方面的最新进展,我们显著提高了配对数据的质量。此外,我们通过突出显示图像中需要根据指令进行更改的部分来增强监督信号。在经过改进的数据上微调后的模型能够比最先进的基线更好地执行细粒度的以对象为中心的编辑,从而缓解了上述问题,自动评估和人工评估均证明了这一点。此外,我们的模型能够泛化到训练期间未见过的领域,例如视觉隐喻。

英文摘要:

Natural language instructions are a powerful interface for editing the outputs of text-to-image diffusion models. However, several challenges need to be addressed: 1) underspecification (the need to model the implicit meaning of instructions) 2) grounding (the need to localize where the edit has to be performed), 3) faithfulness (the need to preserve the elements of the image not affected by the edit instruction). Current approaches focusing on image editing with natural language instructions rely on automatically generated paired data, which, as shown in our investigation, is noisy and sometimes nonsensical, exacerbating the above issues. Building on recent advances in segmentation, Chain-of-Thought prompting, and visual question answering, we significantly improve the quality of the paired data. In addition, we enhance the supervision signal by highlighting parts of the image that need to be changed by the instruction. The model fine-tuned on the improved data is capable of performing fine-grained object-centric edits better than state-of-the-art baselines, mitigating the problems outlined above, as shown by automatic and human evaluations. Moreover, our model is capable of generalizing to domains unseen during training, such as visual metaphors.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑