As the field of Graph Neural Networks (GNN) continues to grow, it experiences a corresponding increase in the need for large, real-world datasets to train and test new GNN models on challenging, realistic problems. Unfortunately, such graph datasets are often generated from online, highly privacy-restricted ecosystems, which makes research and development on these datasets hard, if not impossible. This greatly reduces the amount of benchmark graphs available to researchers, causing the field to rely only on a handful of publicly-available datasets. To address this problem, we introduce a novel graph generative model, Computation Graph Transformer (CGT) that learns and reproduces the distribution of real-world graphs in a privacy-controlled way. More specifically, CGT (1) generates effective benchmark graphs on which GNNs show similar task performance as on the source graphs, (2) scales to process large-scale graphs, (3) incorporates off-the-shelf privacy modules to guarantee end-user privacy of the generated graph. Extensive experiments across a vast body of graph generative models show that only our model can successfully generate privacy-controlled, synthetic substitutes of large-scale real-world graphs that can be effectively used to benchmark GNN models.
B-XAIC Dataset: Benchmarking Explainable AI for Graph Neural Networks Using Chemical Data
University of Edinburgh(爱丁堡大学);Jagiellonian University(雅盖隆大学);Jagiellonian University, Faculty of Mathematics and Computer Science(雅盖隆大学数学与计算机科学学院);Jagiellonian University, Faculty of Chemistry(雅盖隆大学化学学院);Ardigen SA(Ardigen公司)
Benchmarking Graph Neural Networks in Solving Hard Constraint Satisfaction Problems
Department of Computing Sciences, Bocconi University(博科尼大学计算机科学系);Bocconi Institute for Data Science and Analytics (BIDSA)(博科尼数据科学与分析研究所);Department of Physics and Astronomy and INFN, University of Florence(佛罗伦萨大学物理与天文学系及意大利国家核物理研究所);Dipartimento di Fisica, Sapienza Università di Roma(罗马萨皮恩扎大学物理系);CNR - Nanotec, unità di Roma, P.le Aldo Moro 5, 00185 Rome, Italy(意大利罗马国家科研委员会纳米技术单元,Aldo Moro大道5号,00185罗马,意大利);Department of Theoretical Physics, Physics Faculty, University of Havana(古巴哈瓦那大学理论物理系);INFN, sezione di Roma1, P.le Aldo Moro 5, 00185 Rome, Italy(意大利罗马1研究所,Aldo Moro大道5号,00185罗马,意大利)
Protein-SE(3): Benchmarking SE(3)-based Generative Models for Protein Structure Design
School of Computer Science and Technology, East China Normal University(东华师范大学计算机科学与技术学院);Shanghai Institute of AI for Education, East China Normal University(东华师范大学上海人工智能教育研究所);AI Lab, Research Center for Industries of the Future, Westlake University(西湖大学未来产业研究中心人工智能实验室)
Parameterized Argumentation-based Reasoning Tasks for Benchmarking Generative Language Models
Bernoulli Institute of Mathematics, Computer Science and Artificial Intelligence, University of Groningen(伯努利数学、计算机科学和人工智能研究所,格罗宁根大学);Department of Information and Computing Sciences, Utrecht University(信息与计算科学系,乌得勒支大学)
Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting
School of Computer Science, South China Normal University(华南师范大学计算机科学学院);School of Information Technology in Education, South China Normal University(华南师范大学教育信息技术学院)
Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models
Insilico Medicine Hong Kong Ltd;Chemistry and Chemical Biology Department, Harvard University(哈佛大学化学与化学生物学系);Neuromation OU;Hit discovery, Discovery Sciences, Biopharmaceutics R&D, AstraZeneca Gothenburg(阿斯利康哥德堡公司发现科学部药物研发中心);National Research University Higher School of Economics(俄罗斯国家研究型高等经济大学);Department of Chemistry and Department of Computer Science, University of Toronto(多伦多大学化学系与计算机科学系);National Research University Higher School of Economics, St. Petersburg(俄罗斯国家研究型高等经济大学圣彼得堡分校)
AtomBench: A Benchmarking Framework for Generative Crystal Reconstruction Models in Conventional Superconductors
Department of Physics and Astronomy, West Virginia University(物理与天文学系,西弗吉尼亚大学);Department of Materials Science and Engineering, Whiting School of Engineering, The Johns Hopkins University(材料科学与工程系,韦廷工程学院,约翰霍普金斯大学)
LOB-Bench: Benchmarking Generative AI for Finance -- an Application to Limit Order Book Data
Oxford-Man Institute of Quantitative Finance, University of Oxford(牛津大学量化金融研究所);Department of Computer Science, University of Oxford(牛津大学计算机科学系);Department of Statistics, University of Oxford(牛津大学统计学系);Foerster Lab for AI Research, University of Oxford(福尔斯特人工智能研究实验室);J.P. Morgan AI Research(摩根大通人工智能研究)
pmuBAGE: The Benchmarking Assortment of Generated PMU Data for Power System Events
介绍了第一个数据驱动的电力系统事件数据生成模型pmuGE(phasor measurement unit Generator of Events)。我们已经在数千个实际事件上训练了这个模型,并创建了一个名为pmuBAGE(生成的PMU事件的基准分类)的数据集。该数据集由近1000个标记事件数据实例组成,以鼓励对相量测量单元(PMU)数据分析进行基准评估。PMU数据很难获得,特别是那些涵盖事件周期的数据。然而,电力系统问题最近通过数据驱动的机器学习解决方案取得了惊人的进步。一个高度可访问的标准基准数据集将大大加速该领域成功的机器学习技术的发展。提出了一种基于电力系统事件参与分解的PMU数据学习方法,使系统异常时PMU数据生成模型的学习成为可能。该模型可以创建高度真实的事件数据,而不会损害用于训练它的PMU的差异隐私。该数据集可供任何研究人员或从业者在线使用,请访问pmuBAGE Github存储库:https://github.com/NanpengYu/pmuBAGE。
How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers
Department of Computer Science, UCLA(加州大学洛杉矶分校计算机科学系);Department of Computer Science, National University of Singapore(新加坡国立大学计算机科学系);Google Research(谷歌研究院)