Training Data Detection (TDD) is a task aimed at determining whether a specific data instance is used to train a machine learning model. In the computer security literature, TDD is also referred to as Membership Inference Attack (MIA). Given its potential to assess the risks of training data breaches, ensure copyright authentication, and verify model unlearning, TDD has garnered significant attention in recent years, leading to the development of numerous methods. Despite these advancements, there is no comprehensive benchmark to thoroughly evaluate the effectiveness of TDD methods. In this work, we introduce TDDBench, which consists of 13 datasets spanning three data modalities: image, tabular, and text. We benchmark 21 different TDD methods across four detection paradigms and evaluate their performance from five perspectives: average detection performance, best detection performance, memory consumption, and computational efficiency in both time and memory. With TDDBench, researchers can identify bottlenecks and areas for improvement in TDD algorithms, while practitioners can make informed trade-offs between effectiveness and efficiency when selecting TDD algorithms for specific use cases. Our extensive experiments also reveal the generally unsatisfactory performance of TDD algorithms across different datasets. To enhance accessibility and reproducibility, we open-source TDDBench for the research community at https://github.com/zzh9568/TDDBench.
Xi’an Jiaotong University(西安交通大学);Singapore Management University(新加坡管理大学);University of Southampton(南安普顿大学);Harbin Institute of Technology(哈尔滨工业大学);Pengcheng Laboratory(鹏城实验室)
BOND: Benchmarking Unsupervised Outlier Node Detection on Static Attributed Graphs
University of Illinois Chicago(伊利诺伊大学芝加哥分校);Visa Research(Visa研究院);Carnegie Mellon University(卡内基梅隆大学);Beihang University(北京航空航天大学);Arizona State University(亚利桑那州立大学);Illinois Institute of Technology(伊利诺伊理工学院);Lehigh University(理海大学);University of Virginia(弗吉尼亚大学)
A Framework for Evaluating and Benchmarking Concept Drift Detection Methods
University of Coimbra(科英布拉大学);Victoria University of Wellington(惠灵顿维多利亚大学);Commerzbank(德国商业银行);University of Waikato(怀卡托大学);AI Institute, University of Waikato(怀卡托大学人工智能研究所)
A Benchmark for Early-stage Parkinson's Disease Detection from Speech
Centre for Language Studies, Radboud University, Nijmegen, the Netherlands(语言研究所以及拉德堡德大学,尼姆egen,荷兰);Center of Expertise for Parkinson and Movement Disorders, Radboud University Medical Center, Nijmegen, the Netherlands(帕金森及运动障碍专家研究所,拉德堡德大学医学中心,尼姆egen,荷兰)