VDC:基于多模态大语言模型的视觉-语言不一致性的通用数据清洗器
VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Models
AI总结:
本文提出通用数据清洗器VDC,利用多模态大语言模型检测图像与标签间的视觉-语言不一致性,以统一处理投毒样本和噪声标签等各类脏样本,实验验证了其优越的泛化性能。
摘要:
在以数据为中心的人工智能这一新兴概念的推动下,数据在构建AI系统中的作用近来受到高度重视。然而,在现实世界中,数据集可能包含脏样本,例如来自后门攻击的投毒样本、众包场景中的噪声标签,甚至二者的混合。这些脏样本的存在使得深度神经网络变得脆弱且不可靠。因此,检测脏样本对于提升数据集的质量和可靠性至关重要。现有检测器仅专注于检测投毒样本或噪声标签,在处理来自其他领域的脏样本时往往泛化能力较弱。本文发现各类脏样本的一个共同点是图像与关联标签之间的视觉-语言不一致性。为了捕捉模态间的语义不一致性,我们提出了通用数据清洗器(VDC),利用多模态大语言模型(MLLM)在跨模态对齐和推理方面的卓越能力。VDC由三个连续模块组成:视觉问题生成模块,用于生成关于图像的有洞察力的问题;视觉问答模块,通过使用MLLM回答问题来获取视觉内容的语义;随后是视觉答案评估模块,用于评估不一致性。大量实验证明了其在各类别和各类脏样本上的优越性能和泛化能力。代码可在\url{https://github.com/zihao-ai/vdc}获取。
英文摘要:
The role of data in building AI systems has recently been emphasized by the emerging concept of data-centric AI. Unfortunately, in the real-world, datasets may contain dirty samples, such as poisoned samples from backdoor attack, noisy labels in crowdsourcing, and even hybrids of them. The presence of such dirty samples makes the DNNs vunerable and unreliable.Hence, it is critical to detect dirty samples to improve the quality and realiability of dataset. Existing detectors only focus on detecting poisoned samples or noisy labels, that are often prone to weak generalization when dealing with dirty samples from other domains.In this paper, we find a commonality of various dirty samples is visual-linguistic inconsistency between images and associated labels. To capture the semantic inconsistency between modalities, we propose versatile data cleanser (VDC) leveraging the surpassing capabilities of multimodal large language models (MLLM) in cross-modal alignment and reasoning.It consists of three consecutive modules: the visual question generation module to generate insightful questions about the image; the visual question answering module to acquire the semantics of the visual content by answering the questions with MLLM; followed by the visual answer evaluation module to evaluate the inconsistency.Extensive experiments demonstrate its superior performance and generalization to various categories and types of dirty samples. The code is available at \url{https://github.com/zihao-ai/vdc}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。