重新审视大型视觉-语言模型中的数据审计
Revisiting Data Auditing in Large Vision-Language Models
AI总结：
本文发现当前视觉-语言模型（VLMs）的成员推理（MI）基准存在分布偏移导致性能虚高，构建了无偏新基准并证明现有方法失效，同时揭示了MI的理论局限并指出三种可行的审计场景。
摘要：
随着大型语言模型（LLMs）的兴起，大型视觉-语言模型（VLMs）——将视觉编码器与LLMs集成以实现准确的视觉定位——在通用智能体和机器人控制等任务中展现出巨大潜力。然而，VLMs通常在海量网络抓取的图像上进行训练，引发了对版权侵权和隐私侵犯的担忧，使得数据审计变得日益紧迫。成员推理（MI）用于判定某个样本是否被用于训练，已成为一种关键的审计技术，在LLaVA等开源VLMs上取得了可喜的成果（AUC > 80%）。在本研究中，我们重新审视这些进展并揭示了一个关键问题：当前的MI基准在成员与非成员图像之间存在分布偏移，引入了虚高MI性能的捷径线索。我们进一步分析了这些偏移的本质，并提出了一种基于最优传输的原则性度量方法来量化分布差异。为了在现实环境中评估MI，我们构建了具有独立同分布（i.i.d.）成员与非成员图像的新基准。现有的MI方法在这些无偏条件下失效，表现仅略优于随机猜测。此外，我们通过探查VLM嵌入空间内的贝叶斯最优性来探索MI的理论上界，发现不可约错误率依然很高。尽管前景悲观，我们分析了VLMs的MI为何尤为困难，并确定了三种审计变得可行的实际场景——微调、获取真实文本以及基于集合的推理。我们的研究提出了VLMs MI局限与机遇的系统性视角，为未来可信数据审计工作提供了指导。
英文摘要：
With the surge of large language models (LLMs), Large Vision-Language Models (VLMs)--which integrate vision encoders with LLMs for accurate visual grounding--have shown great potential in tasks like generalist agents and robotic control. However, VLMs are typically trained on massive web-scraped images, raising concerns over copyright infringement and privacy violations, and making data auditing increasingly urgent. Membership inference (MI), which determines whether a sample was used in training, has emerged as a key auditing technique, with promising results on open-source VLMs like LLaVA (AUC > 80%). In this work, we revisit these advances and uncover a critical issue: current MI benchmarks suffer from distribution shifts between member and non-member images, introducing shortcut cues that inflate MI performance. We further analyze the nature of these shifts and propose a principled metric based on optimal transport to quantify the distribution discrepancy. To evaluate MI in realistic settings, we construct new benchmarks with i.i.d. member and non-member images. Existing MI methods fail under these unbiased conditions, performing only marginally better than chance. Further, we explore the theoretical upper bound of MI by probing the Bayes Optimality within the VLM's embedding space and find the irreducible error rate remains high. Despite this pessimistic outlook, we analyze why MI for VLMs is particularly challenging and identify three practical scenarios--fine-tuning, access to ground-truth texts, and set-based inference--where auditing becomes feasible. Our study presents a systematic view of the limits and opportunities of MI for VLMs, providing guidance for future efforts in trustworthy data auditing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
