发表机构
School of Computer Science and Technology, Beijing Institute of Technology; School of Computer Science and Engineering, Northeastern University; Faculty of Computer Science, Dalhousie University; King Abdullah University of Science and Technology（北京理工大学计算机科学与技术学院; 东北大学计算机科学与工程学院; 达尔豪斯大学计算机科学学院; 阿卜杜拉国王科技大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究提出MemCatalyst数据投毒工具，通过文本与图像投毒策略增强视觉-语言模型的成员推断审计性能，在黑盒设置下投毒样本可跨架构迁移，仅需少量投毒样本即可提升审计效果且不影响模型性能。
AI 中文摘要
视觉-语言模型（VLMs）之所以能取得出色性能，很大程度上得益于互联网上可用的大量训练数据。与此同时，数据持有者（如艺术家）迫切需要确定其数据是否被未经授权地用于模型训练，这涉及知识产权和个人隐私问题。数据审计，尤其是通过成员推断（MI），已成为一种直接工具受到关注。本研究提出MemCatalyst，一套数据投毒工具，旨在增强VLMs的数据审计性能。MemCatalyst采用两种策略：文本投毒（PT）和图像投毒（PI）。MemCatalyst迫使VLMs在训练期间过度学习图像特征与文本语义之间的特定不一致性，从而提高它们对成员信息审计的敏感性。重要的是，在黑盒设置中，已证明投毒样本在不同VLM架构之间的可迁移性是有效的。使用两个知名VLMs上的五种最先进数据审计方法进行的广泛评估表明，MemCatalyst以极少的投毒样本预算显著提高了MI AUC分数，同时对模型性能的影响可忽略不计。
英文摘要
Vision-Language models (VLMs) achieve outstanding performance largely due to the amount of training data available on the internet. At the same time, data holders (e.g., artists) urgently need to determine whether their data has been used for model training without authorization, which concerns both intellectual property rights and personal privacy. Data auditing, particularly through membership inference (MI), has attracted attention as a direct tool. This work proposes MemCatalyst, a set of data poisoning tools, aiming to amplify the data auditing performance on VLMs. MemCatalyst employs two strategies: Poisoning Text (PT) and Poisoning Image (PI). MemCatalyst forces VLMs to over-learn specific inconsistencies between image features and textual semantics during training, thereby increasing their susceptibility to membership information auditing. Crucially, the transferability of poisoned samples across different VLM architectures is demonstrated to be effective in the black-box setting. Extensive evaluations using five state-of-the-art data audits on two prominent VLMs demonstrate that MemCatalyst markedly enhances MI AUC scores with a minimal budget of poisoned samples, while maintaining a negligible impact on model performance.
