使用多模态语义扰动检测VLMs中的污染
Contamination Detection for VLMs using Multi-Modal Semantic Perturbation
AI总结：
本文提出了一种基于多模态语义扰动的检测方法，用于识别和验证受污染的视觉语言模型。
摘要：
近年来，视觉语言模型（VLMs）在多个基准任务上取得了最先进的性能。然而，使用大规模互联网、通常为专有性质的预训练语料库引发了从业者和用户的关键担忧：由于测试集泄漏导致的性能虚高。尽管先前的工作提出了缓解策略，如预训练数据去污染和LLM的基准重设，但开发用于检测污染VLMs的方法的互补方向仍被忽视。为解决这一差距，我们故意在流行的基准上污染开源VLMs，并展示现有检测方法要么彻底失败，要么表现出不一致的行为。然后，我们提出了一种新颖的简单而有效的检测方法，基于多模态语义扰动，证明污染模型在受控扰动下无法泛化。最后，我们验证了我们的方法在多种现实污染策略下的表现，确认了其鲁棒性和有效性。代码和扰动数据集发布在https://github.com/jadenpark0/mm-perturb。
英文摘要：
Recent advances in Vision-Language Models (VLMs) have achieved state-of-the-art performance on numerous benchmark tasks. However, the use of internet-scale, often proprietary, pretraining corpora raises a critical concern for both practitioners and users: inflated performance due to test-set leakage. While prior works have proposed mitigation strategies such as decontamination of pretraining data and benchmark redesign for LLMs, the complementary direction of developing detection methods for contaminated VLMs remains underexplored. To address this gap, we deliberately contaminate open-source VLMs on popular benchmarks and show that existing detection approaches either fail outright or exhibit inconsistent behavior. We then propose a novel simple yet effective detection method based on multi-modal semantic perturbation, demonstrating that contaminated models fail to generalize under controlled perturbations. Finally, we validate our approach across multiple realistic contamination strategies, confirming its robustness and effectiveness. The code and perturbed dataset are released at https://github.com/jadenpark0/mm-perturb.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
