利用提示工程通过 LLM 进行廉价伪造检测
Cheap-fake Detection with LLM using Prompt Engineering
AI总结:
本文面向 ICME'23 廉价伪造检测挑战,在 COSMOS 基线中引入基于提示工程的 GPT3.5 特征提取模块,以建模两条图像说明的相关性并提升 OOC 媒体检测性能。
摘要:
在新闻中滥用真实照片并配以相互矛盾的图像说明,是媒体脱离语境(out-of-context,OOC)滥用的一个例子。为了检测 OOC 媒体,人们必须判断陈述的准确性,并评估三元组(即图像和两条说明)是否与同一事件相关。本文提出了一种新颖的可学习方法,用于在 ICME'23 Detecting Cheapfakes 大挑战赛中检测 OOC 媒体。所提方法基于 COSMOS 结构,该结构评估图像与说明之间以及两条说明之间的一致性。我们通过引入大型语言模型(LLM)GPT3.5 作为特征提取器来增强基线算法。具体而言,我们提出了一种创新的特征提取方法,利用提示工程来开发一个稳健且可靠的 GPT3.5 模型特征提取器。所提方法捕捉两条说明之间的相关性,并将该模块有效集成到 COSMOS 基线模型中,从而能够更深入地理解说明之间的关系。通过引入该模块,我们证明了廉价伪造检测性能显著提升的潜力。所提方法对自然语言处理、图像说明和文本到图像合成等多种应用具有积极意义。用于提交的 Docker 可在 https://hub.docker.com/repository/docker/mulns/acmmmcheapfakes 获取。
英文摘要:
The misuse of real photographs with conflicting image captions in news items is an example of the out-of-context (OOC) misuse of media. In order to detect OOC media, individuals must determine the accuracy of the statement and evaluate whether the triplet (~\textit{i.e.}, the image and two captions) relates to the same event. This paper presents a novel learnable approach for detecting OOC media in ICME'23 Grand Challenge on Detecting Cheapfakes. The proposed method is based on the COSMOS structure, which assesses the coherence between an image and captions, as well as between two captions. We enhance the baseline algorithm by incorporating a Large Language Model (LLM), GPT3.5, as a feature extractor. Specifically, we propose an innovative approach to feature extraction utilizing prompt engineering to develop a robust and reliable feature extractor with GPT3.5 model. The proposed method captures the correlation between two captions and effectively integrates this module into the COSMOS baseline model, which allows for a deeper understanding of the relationship between captions. By incorporating this module, we demonstrate the potential for significant improvements in cheap-fakes detection performance. The proposed methodology holds promising implications for various applications such as natural language processing, image captioning, and text-to-image synthesis. Docker for submission is available at https://hub.docker.com/repository/docker/mulns/ acmmmcheapfakes.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。