arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.19119cs.CVcs.CL
收藏

VIXEN:用于图像差异描述的可视文本比较网络

VIXEN: Visual Text Comparison Network for Image Difference Captioning

University of Surrey(萨里大学);Adobe Research(Adobe研究院)

Alexander Black, Jing Shi, Yifei Fan, Tu Bui, John Collomosse

AI总结:

VIXEN通过成对线性映射图像特征为语言模型生成软提示,利用合成数据训练,以文本描述图像差异,对抗篡改图像传播的误导信息。

摘要:

我们提出了VIXEN——一种用文本简洁地总结一对图像之间视觉差异的技术,以突出其中存在的任何内容篡改。我们提出的网络以成对方式线性映射图像特征,为预训练的大型语言模型构建软提示。我们通过使用来自最近的InstructPix2Pix数据集(通过提示到提示编辑框架生成)的合成篡改图像进行训练,解决了现有图像差异描述(IDC)数据集中训练数据量少和篡改类型缺乏多样性的挑战。我们使用GPT-3生成的变更摘要来扩充该数据集。我们表明,VIXEN能为多样的图像内容和编辑类型生成最先进且易于理解的差异描述,为缓解通过篡改图像内容传播的错误信息提供了潜在手段。代码和数据可在http://github.com/alexblck/vixen获取。

英文摘要:

We present VIXEN - a technique that succinctly summarizes in text the visual differences between a pair of images in order to highlight any content manipulation present. Our proposed network linearly maps image features in a pairwise manner, constructing a soft prompt for a pretrained large language model. We address the challenge of low volume of training data and lack of manipulation variety in existing image difference captioning (IDC) datasets by training on synthetically manipulated images from the recent InstructPix2Pix dataset generated via prompt-to-prompt editing framework. We augment this dataset with change summaries produced via GPT-3. We show that VIXEN produces state-of-the-art, comprehensible difference captions for diverse image contents and edit types, offering a potential mitigation against misinformation disseminated via manipulated image content. Code and data are available at http://github.com/alexblck/vixen

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑