Synthesizing relational data has started to receive more attention from researchers, practitioners, and industry. The task is more difficult than synthesizing a single table due to the added complexity of relationships between tables. For the same reason, benchmarking methods for synthesizing relational data introduces new challenges. Our work is motivated by a lack of an empirical evaluation of state-of-the-art methods and by gaps in the understanding of how such an evaluation should be done. We review related work on relational data synthesis, common benchmarking datasets, and approaches to measuring the fidelity and utility of synthetic data. We combine the best practices and a novel robust detection approach into a benchmarking tool and use it to compare six methods, including two commercial tools. While some methods are better than others, no method is able to synthesize a dataset that is indistinguishable from original data. For utility, we typically observe moderate correlation between real and synthetic data for both model predictive performance and feature importance.
ContextGNN goes to Elliot: Towards Benchmarking Relational Deep Learning for Static Link Prediction (aka Personalized Item Recommendation)
University of Barcelona（巴塞罗那大学）；University of Southern California（南加州大学）；Université Paris-Saclay（巴黎萨克雷大学）；CentraleSupélec（中央理工-高等电力学院）；Inria（法国国家信息与自动化研究所）
4DBInfer: A 4D Benchmarking Toolbox for Graph-Centric Predictive Modeling on Relational DBs
Amazon Web Services（亚马逊网络服务）；Shanghai Jiaotong University（上海交通大学）；The Hong Kong University of Science and Technology（香港科技大学）；Fudan University（复旦大学）；Tsinghua University（清华大学）；Southern University of Science and Technology（南方科技大学）；Peking University（北京大学）；The Hong Kong Polytechnic University（香港理工大学）；Carnegie Mellon University（卡内基梅隆大学）
A reproducible and extensible framework for benchmarking competing risks survival models
Cancer Research UK Scotland Centre（英国癌症研究苏格兰中心）；Institute of Genetics and Cancer（遗传与癌症研究所）；University of Edinburgh（爱丁堡大学）；University of Augsburg（奥格斯堡大学）
Benchmarking Deep Learning-Based Reconstruction Methods for Photoacoustic Computed Tomography with Clinically Relevant Synthetic Datasets
Department of Bioengineering, University of Illinois Urbana-Champaign（生物工程系，伊利诺伊大学厄巴纳-香槟分校）；Department of Electrical and Computer Engineering, University of Illinois Urbana-Champaign（电气与计算机工程系，伊利诺伊大学厄巴纳-香槟分校）；Department of Biomedical Engineering, The University of Texas at Austin（生物医学工程系，德克萨斯大学奥斯汀分校）；Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin（奥登计算工程与科学研究所，德克萨斯大学奥斯汀分校）
GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Shanghai Jiao Tong University（上海交通大学）；King Abdullah University of Science and Technology (KAUST)（卡尔斯鲁厄大学）；Tsinghua University（清华大学）；National University of Singapore（新加坡国立大学）
Generating Synthetic Electronic Health Record Data: a Methodological Scoping Review with Benchmarking on Phenotype Data and Open-Source Software
Department of Biostatistics, University of Michigan（生物统计学系，密歇根大学）；Department of Biomedical Informatics and Data Science, Yale University（生物医学信息学与数据科学系，耶鲁大学）；Department of Biostatistics, Yale University（生物统计学系，耶鲁大学）
Synthetic Data Generation for Brain-Computer Interfaces: Overview, Benchmarking, and Future Directions
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China（华中科技大学人工智能与自动化学院，武汉，中国）；Zhongguancun Academy, Beijing, China（中关村学院，北京，中国）
BioCoref: Benchmarking Biomedical Coreference Resolution with LLMs
Computational Bioscience Program University of Colorado Anschutz Medical Campus（科学生物学程序，科罗拉多大学安舒茨医学校区）；Department of Pediatrics University of Chicago（儿科学系，芝加哥大学）
Benchmarking Automated Clinical Language Simplification: Dataset, Algorithm, and Evaluation
College of Information Sciences and Technology, Pennsylvania State University（宾夕法尼亚州立大学信息科学与技术学院）；School of Software Technology, Dalian University of Technology（大连理工大学软件学院）；Relativity（相对论公司）
Benchmarking Time-localized Explanations for Audio Classification Models
UBAArgentina（UBA阿根廷）；CONICET-UBAArgentina（CONICET-UBA阿根廷）；Departamento de Computación, FCEyN（计算系）；Instituto de Investigación en Ciencias de la Computación (ICC)（计算机科学研究所）
The Public Health and Environmental Surveillance Open Data Model (PHES-ODM) Version 3: An Open, Relational Data Model and Interoperability Framework for Wastewater Surveillance
Ottawa Hospital Research Institute, University of Ottawa（渥太华医院研究所，渥太华大学）；Ghent University Faculty of Bioscience Engineering（根特大学生物工程学院）；ModelEAU, Université Laval（ModelEAU，拉瓦尔大学）；Public Health Agency of Canada National Microbiology Laboratory（加拿大公共卫生局国家微生物实验室）；European Commission, Joint Research Centre（欧盟委员会联合研究中心）
