arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.17652cs.LGcs.SE
收藏

面向深度学习测试输入生成的生成式AI模型基准测试

Benchmarking Generative AI Models for Deep Learning Test Input Generation

University of Udine(乌迪内大学);Università della Svizzera italiana(瑞士意大利语大学);Technical University of Munich(慕尼黑工业大学);fortiss(弗劳恩霍夫应用信息技术研究所(fortiss))

Maryam, Matteo Biagiola, Andrea Stocco, Vincenzo Riccio

AI总结:

本文对生成式AI模型与测试输入生成器结合的效果进行基准测试,发现简单架构适用于MNIST等简单数据集,而Diffusion Models等复杂架构在ImageNet等复杂数据集上表现更优。

摘要:

测试输入生成器(TIGs)对于评估深度学习(DL)图像分类器在训练集和测试集之外的输入上提供正确预测的能力至关重要。生成式AI(GenAI)模型的最新进展使其成为创建和操作合成图像的强大工具,尽管这些进展也意味着训练的复杂性和资源需求增加。在这项工作中,我们对不同的GenAI模型进行基准测试,并将其与TIGs结合,从领域有效性和标签保留的角度评估所生成测试图像的有效性、效率和质量。我们进行了一项实证研究,涉及三种不同的GenAI架构(VAEs、GANs、Diffusion Models)、五个复杂度递增的分类任务以及364次人类评估。结果表明,对于像MNIST这样较不复杂的数据集,VAEs等较简单的架构就足够了。然而,在处理像ImageNet这样特征丰富的数据集时,Diffusion Models等更复杂的架构通过生成更多有效且能引发误分类的输入,实现了更优的性能。

英文摘要:

Test Input Generators (TIGs) are crucial to assess the ability of Deep Learning (DL) image classifiers to provide correct predictions for inputs beyond their training and test sets. Recent advancements in Generative AI (GenAI) models have made them a powerful tool for creating and manipulating synthetic images, although these advancements also imply increased complexity and resource demands for training. In this work, we benchmark and combine different GenAI models with TIGs, assessing their effectiveness, efficiency, and quality of the generated test images, in terms of domain validity and label preservation. We conduct an empirical study involving three different GenAI architectures (VAEs, GANs, Diffusion Models), five classification tasks of increasing complexity, and 364 human evaluations. Our results show that simpler architectures, such as VAEs, are sufficient for less complex datasets like MNIST. However, when dealing with feature-rich datasets, such as ImageNet, more sophisticated architectures like Diffusion Models achieve superior performance by generating a higher number of valid, misclassification-inducing inputs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑