arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.03363cs.CRcs.LG
收藏

TDDBench:一个用于训练数据检测的基准

TDDBench: A Benchmark for Training data detection

University of Science and Technology of China(中国科学技术大学);The Hong Kong University of Science and Technology(香港科技大学)

Zhihao Zhu, Yi Yang, Defu Lian

AI总结:

本文提出TDDBench,一个涵盖3种模态13个数据集的全面基准,在4种检测范式下评估21种TDD方法,从5个维度分析其性能,揭示现有算法的不足,促进TDD研究与应用。

摘要:

训练数据检测(TDD)是一项旨在确定特定数据实例是否被用于训练机器学习模型的任务。在计算机安全文献中,TDD也被称为成员推理攻击(MIA)。鉴于其在评估训练数据泄露风险、确保版权认证以及验证模型遗忘方面的潜力,TDD近年来引起了广泛关注,并催生了大量方法。尽管取得了这些进展,目前仍缺乏一个全面的基准来彻底评估TDD方法的有效性。在本研究中,我们提出了TDDBench,它包含跨越三种数据模态的13个数据集:图像、表格和文本。我们在四种检测范式下对21种不同的TDD方法进行了基准测试,并从五个角度评估了它们的性能:平均检测性能、最佳检测性能、内存消耗,以及时间和内存方面的计算效率。借助TDDBench,研究人员能够识别TDD算法中的瓶颈和改进空间,而从业者则可以在为特定用例选择TDD算法时,在有效性和效率之间做出明智的权衡。我们广泛的实验还揭示了TDD算法在不同数据集上普遍不令人满意的性能。为了增强可访问性和可重复性,我们在https://github.com/zzh9568/TDDBench向研究社区开源了TDDBench。

英文摘要:

Training Data Detection (TDD) is a task aimed at determining whether a specific data instance is used to train a machine learning model. In the computer security literature, TDD is also referred to as Membership Inference Attack (MIA). Given its potential to assess the risks of training data breaches, ensure copyright authentication, and verify model unlearning, TDD has garnered significant attention in recent years, leading to the development of numerous methods. Despite these advancements, there is no comprehensive benchmark to thoroughly evaluate the effectiveness of TDD methods. In this work, we introduce TDDBench, which consists of 13 datasets spanning three data modalities: image, tabular, and text. We benchmark 21 different TDD methods across four detection paradigms and evaluate their performance from five perspectives: average detection performance, best detection performance, memory consumption, and computational efficiency in both time and memory. With TDDBench, researchers can identify bottlenecks and areas for improvement in TDD algorithms, while practitioners can make informed trade-offs between effectiveness and efficiency when selecting TDD algorithms for specific use cases. Our extensive experiments also reveal the generally unsatisfactory performance of TDD algorithms across different datasets. To enhance accessibility and reproducibility, we open-source TDDBench for the research community at https://github.com/zzh9568/TDDBench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑