arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.04756cs.CV
收藏

基于软逻辑正则化的可解释多模态上下文错位检测

Interpretable Multimodal Out-of-context Detection with Soft Logic Regularization

School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Center for Research on Intelligent Perception and Computing (CRIPAC)(智能感知与计算研究中心);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)

Huanhuan Ma, Jinghao Zhang, Qiang Liu, Shu Wu, Liang Wang

AI总结:

本文提出LOGRAN方法,通过短语级潜变量和逻辑规则聚合实现可解释的图像-文本上下文错位检测,并在NewsCLIPpings数据集上验证了其竞争性性能与解释能力。

摘要:

信息通过移动设备和媒体迅速传播,导致虚假或欺骗性新闻广泛扩散,引发社会的重大关切。在不同类型的错误信息中,图像再利用(也称为上下文错位错误信息)仍然高度普遍且有效。然而,当前检测上下文错位错误信息的方法往往缺乏可解释性,提供的解释有限。在本研究中,我们提出了一种用于上下文错位检测的逻辑正则化方法,称为LOGRAN(LOGic Regularization for out-of-context ANalysis,上下文错位分析的逻辑正则化)。LOGRAN的主要目标是在短语级别分解上下文错位检测。通过采用潜变量进行短语级预测,图像-文本对的最终预测可以利用逻辑规则进行聚合。潜变量还为最终结果的推导方式提供了解释,使得这种细粒度检测方法天然具有可解释性。我们在NewsCLIPpings数据集上评估了LOGRAN的性能,展示了具有竞争力的整体结果。可视化示例还揭示了上下文错位图像在短语级别上的忠实预测,并附有解释。这凸显了我们的方法在解决上下文错位检测和增强可解释性方面的有效性。

英文摘要:

The rapid spread of information through mobile devices and media has led to the widespread of false or deceptive news, causing significant concerns in society. Among different types of misinformation, image repurposing, also known as out-of-context misinformation, remains highly prevalent and effective. However, current approaches for detecting out-of-context misinformation often lack interpretability and offer limited explanations. In this study, we propose a logic regularization approach for out-of-context detection called LOGRAN (LOGic Regularization for out-of-context ANalysis). The primary objective of LOGRAN is to decompose the out-of-context detection at the phrase level. By employing latent variables for phrase-level predictions, the final prediction of the image-caption pair can be aggregated using logical rules. The latent variables also provide an explanation for how the final result is derived, making this fine-grained detection method inherently explanatory. We evaluate the performance of LOGRAN on the NewsCLIPpings dataset, showcasing competitive overall results. Visualized examples also reveal faithful phrase-level predictions of out-of-context images, accompanied by explanations. This highlights the effectiveness of our approach in addressing out-of-context detection and enhancing interpretability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑