arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.18033cs.CV
收藏

使用扩散模型揭示文本-图像不一致性

Exposing Text-Image Inconsistency Using Diffusion Models

University at Buffalo, State University of New York(纽约州立大学布法罗分校)

Mingzhen Huang, Shan Jia, Zhou Zhou, Yan Ju, Jialing Cai, Siwei Lyu

AI总结:

针对文本-图像不一致性识别缺乏可解释性的问题,提出基于扩散模型的D-TIIL方法,通过文本嵌入和修改图像区域定位并可视化语义不一致,并构建1.4万对数据的TIIL数据集进行评估,提供了可扩展且基于证据的打击虚假信息框架。

摘要:

在与广泛的网络虚假信息作斗争的过程中,文本-图像不一致性成为一个日益严重的问题,即图像被误导性地与具有不同意图或含义的文本配对。现有的基于分类的文本-图像不一致性方法能够识别上下文的不一致,但无法为人类提供可理解的、可解释的决策依据。尽管人工评估更加细致,但在大规模应用中不切实际且容易出错。为了解决这些局限性,本研究引入了 D-TIIL(基于扩散的文本-图像不一致性定位),它采用文本到图像的扩散模型来定位文本和图像对中的语义不一致。这些在大规模数据集上训练的模型充当“全知”代理,过滤掉无关信息并结合背景知识来识别不一致性。此外,D-TIIL 使用文本嵌入和修改后的图像区域来可视化这些不一致性。为了评估 D-TIIL 的有效性,我们引入了一个包含 1.4 万对一致和不一致文本-图像对的新 TIIL 数据集。与现有数据集不同,TIIL 能够在单个单词和图像区域级别进行评估,并经过精心设计以表示各种不一致性。D-TIIL 提供了一种可扩展且基于证据的方法来识别和定位文本-图像不一致性,为未来打击虚假信息的研究提供了一个稳健的框架。

英文摘要:

In the battle against widespread online misinformation, a growing problem is text-image inconsistency, where images are misleadingly paired with texts with different intent or meaning. Existing classification-based methods for text-image inconsistency can identify contextual inconsistencies but fail to provide explainable justifications for their decisions that humans can understand. Although more nuanced, human evaluation is impractical at scale and susceptible to errors. To address these limitations, this study introduces D-TIIL (Diffusion-based Text-Image Inconsistency Localization), which employs text-to-image diffusion models to localize semantic inconsistencies in text and image pairs. These models, trained on large-scale datasets act as ``omniscient" agents that filter out irrelevant information and incorporate background knowledge to identify inconsistencies. In addition, D-TIIL uses text embeddings and modified image regions to visualize these inconsistencies. To evaluate D-TIIL's efficacy, we introduce a new TIIL dataset containing 14K consistent and inconsistent text-image pairs. Unlike existing datasets, TIIL enables assessment at the level of individual words and image regions and is carefully designed to represent various inconsistencies. D-TIIL offers a scalable and evidence-based approach to identifying and localizing text-image inconsistency, providing a robust framework for future research combating misinformation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑