Network Intrusion Detection Systems (NIDSs) are an increasingly important tool for the prevention and mitigation of cyber attacks. A number of labelled synthetic datasets generated have been generated and made publicly available by researchers, and they have become the benchmarks via which new ML-based NIDS classifiers are being evaluated. Recently published results show excellent classification performance with these datasets, increasingly approaching 100 percent performance across key evaluation metrics such as accuracy, F1 score, etc. Unfortunately, we have not yet seen these excellent academic research results translated into practical NIDS systems with such near-perfect performance. This motivated our research presented in this paper, where we analyse the statistical properties of the benign traffic in three of the more recent and relevant NIDS datasets, (CIC, UNSW, ...). As a comparison, we consider two datasets obtained from real-world production networks, one from a university network and one from a medium size Internet Service Provider (ISP). Our results show that the two real-world datasets are quite similar among themselves in regards to most of the considered statistical features. Equally, the three synthetic datasets are also relatively similar within their group. However, and most importantly, our results show a distinct difference of most of the considered statistical features between the three synthetic datasets and the two real-world datasets. Since ML relies on the basic assumption of training and test datasets being sampled from the same distribution, this raises the question of how well the performance results of ML-classifiers trained on the considered synthetic datasets can translate and generalise to real-world networks. We believe this is an interesting and relevant question which provides motivation for further research in this space.
Benchmarking Deep Learning-Based Reconstruction Methods for Photoacoustic Computed Tomography with Clinically Relevant Synthetic Datasets
Department of Bioengineering, University of Illinois Urbana-Champaign（生物工程系，伊利诺伊大学厄巴纳-香槟分校）；Department of Electrical and Computer Engineering, University of Illinois Urbana-Champaign（电气与计算机工程系，伊利诺伊大学厄巴纳-香槟分校）；Department of Biomedical Engineering, The University of Texas at Austin（生物医学工程系，德克萨斯大学奥斯汀分校）；Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin（奥登计算工程与科学研究所，德克萨斯大学奥斯汀分校）
SUB: Benchmarking CBM Generalization via Synthetic Attribute Substitutions
Technical University of Munich, Helmholtz Munich, Munich Center for Machine Learning (MCML)（慕尼黑技术大学、亥姆霍兹慕尼黑、慕尼黑机器学习中心）；LTCI, Télécom Paris, Institut Polytechnique de Paris（LTCI、巴黎电信、巴黎理工学院）
Generating Synthetic Electronic Health Record Data: a Methodological Scoping Review with Benchmarking on Phenotype Data and Open-Source Software
Department of Biostatistics, University of Michigan（生物统计学系，密歇根大学）；Department of Biomedical Informatics and Data Science, Yale University（生物医学信息学与数据科学系，耶鲁大学）；Department of Biostatistics, Yale University（生物统计学系，耶鲁大学）
Benchmarking Time Series Generation Methods for Privacy-Preserving Forecasting
University of Minho（米尼奥大学）；Universidade do Porto（波尔图大学）；Laboratory for Artificial Intelligence and Computer Science (LIACC)（人工智能与计算机科学实验室（LIACC））；Fraunhofer Portugal AICOS（弗劳恩霍夫葡萄牙AICOS机构）；INESCTEC；University of Coimbra（科英布拉大学）
Synthetic Data Generation for Brain-Computer Interfaces: Overview, Benchmarking, and Future Directions
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China（华中科技大学人工智能与自动化学院，武汉，中国）；Zhongguancun Academy, Beijing, China（中关村学院，北京，中国）
BOND: Benchmarking Unsupervised Outlier Node Detection on Static Attributed Graphs
University of Illinois Chicago（伊利诺伊大学芝加哥分校）；Visa Research（Visa研究院）；Carnegie Mellon University（卡内基梅隆大学）；Beihang University（北京航空航天大学）；Arizona State University（亚利桑那州立大学）；Illinois Institute of Technology（伊利诺伊理工学院）；Lehigh University（理海大学）；University of Virginia（弗吉尼亚大学）
AFABench: A Generic Framework for Benchmarking Active Feature Acquisition
Chalmers University of Technology \& University of Gothenburg Sweden；Chalmers University of Technology Sweden；Chalmers University of Technology \& University of Gothenburg；Chalmers University of Technology
Cornell University（康奈尔大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Science and Technology of China（中国科学技术大学）；University of Sydney（悉尼大学）；Huazhong University of Science and Technology（华中科技大学）；Microsoft Research（微软研究院）；Texas A&M University（得克萨斯农工大学）
