Test Input Generators (TIGs) are crucial to assess the ability of Deep Learning (DL) image classifiers to provide correct predictions for inputs beyond their training and test sets. Recent advancements in Generative AI (GenAI) models have made them a powerful tool for creating and manipulating synthetic images, although these advancements also imply increased complexity and resource demands for training.
In this work, we benchmark and combine different GenAI models with TIGs, assessing their effectiveness, efficiency, and quality of the generated test images, in terms of domain validity and label preservation. We conduct an empirical study involving three different GenAI architectures (VAEs, GANs, Diffusion Models), five classification tasks of increasing complexity, and 364 human evaluations. Our results show that simpler architectures, such as VAEs, are sufficient for less complex datasets like MNIST. However, when dealing with feature-rich datasets, such as ImageNet, more sophisticated architectures like Diffusion Models achieve superior performance by generating a higher number of valid, misclassification-inducing inputs.
Protein-SE(3): Benchmarking SE(3)-based Generative Models for Protein Structure Design
School of Computer Science and Technology, East China Normal University(东华师范大学计算机科学与技术学院);Shanghai Institute of AI for Education, East China Normal University(东华师范大学上海人工智能教育研究所);AI Lab, Research Center for Industries of the Future, Westlake University(西湖大学未来产业研究中心人工智能实验室)
Parameterized Argumentation-based Reasoning Tasks for Benchmarking Generative Language Models
Bernoulli Institute of Mathematics, Computer Science and Artificial Intelligence, University of Groningen(伯努利数学、计算机科学和人工智能研究所,格罗宁根大学);Department of Information and Computing Sciences, Utrecht University(信息与计算科学系,乌得勒支大学)
AtomBench: A Benchmarking Framework for Generative Crystal Reconstruction Models in Conventional Superconductors
Department of Physics and Astronomy, West Virginia University(物理与天文学系,西弗吉尼亚大学);Department of Materials Science and Engineering, Whiting School of Engineering, The Johns Hopkins University(材料科学与工程系,韦廷工程学院,约翰霍普金斯大学)
Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models
Insilico Medicine Hong Kong Ltd;Chemistry and Chemical Biology Department, Harvard University(哈佛大学化学与化学生物学系);Neuromation OU;Hit discovery, Discovery Sciences, Biopharmaceutics R&D, AstraZeneca Gothenburg(阿斯利康哥德堡公司发现科学部药物研发中心);National Research University Higher School of Economics(俄罗斯国家研究型高等经济大学);Department of Chemistry and Department of Computer Science, University of Toronto(多伦多大学化学系与计算机科学系);National Research University Higher School of Economics, St. Petersburg(俄罗斯国家研究型高等经济大学圣彼得堡分校)
Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting
School of Computer Science, South China Normal University(华南师范大学计算机科学学院);School of Information Technology in Education, South China Normal University(华南师范大学教育信息技术学院)
LOB-Bench: Benchmarking Generative AI for Finance -- an Application to Limit Order Book Data
Oxford-Man Institute of Quantitative Finance, University of Oxford(牛津大学量化金融研究所);Department of Computer Science, University of Oxford(牛津大学计算机科学系);Department of Statistics, University of Oxford(牛津大学统计学系);Foerster Lab for AI Research, University of Oxford(福尔斯特人工智能研究实验室);J.P. Morgan AI Research(摩根大通人工智能研究)
Practical Design and Benchmarking of Generative AI Applications for Surgical Billing and Coding
Duke University Medical Center(杜克大学医学中心);Patient Revenue Management Organization, Duke University Health System(杜克大学卫生系统患者收入管理组织);Duke Pratt School of Engineering(杜克普拉特工程学院);Duke University School of Medicine(杜克大学医学院);MSTAT
Reject Illegal Inputs with Generative Classifier Derived from Any Discriminative Classifier
The University of Hong Kong(香港大学)
生成式分类器已被证明有希望检测非法输入,包括对抗性样本和分布外样本。Supervised Deep Infomax~(SDIM)是一个可扩展的端到端框架,用于学习生成分类器。在本文中,我们提出了一个修改的SDIM称为SDIM- \emph{logit}。SDIM- \emph{logit}不是从头开始训练生成分类器,而是首先将任何给定的判别分类器产生的logit作为输入,并生成logit表示;然后通过对logit表示施加统计约束来导出生成分类器。SDIM- \emph{logit}能够继承判别分类器的性能而不损失。SDIM- \emph{logit}引起的额外参数的数量可以忽略不计,并且可以有效地训练与基本分类器固定。我们执行\emph{classification with rejection},其中类条件小于预先选择的阈值的测试样本将被拒绝而不进行预测。在非法输入(包括对抗样本、常见错误样本和分布外样本)上的实验表明,SDIM- \emph{logit}允许拒绝一部分测试样本,显著提高了在左测试集上的性能.
Machine-learning for photoplethysmography analysis: Benchmarking feature, image, and signal-based approaches
Carl von Ossietzky Universität Oldenburg(奥尔登堡卡尔·冯·奥西特齐大学);Laboratoire national de métrologie et d’essais(国家计量与测试实验室);Biomedical Engineering Institute, Kaunas University of Technology(克莱萨斯理工大学生物医学工程研究所);Physikalisch-Technische Bundesanstalt(德国物理技术部);National Physical Laboratory(国家物理实验室);King’s College London(伦敦国王学院);University of Surrey(萨里大学);Department of Public Health and Primary Care, University of Cambridge(剑桥大学公共卫生与初级保健系)