arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.02870cs.SEcs.CV
收藏

基于变形测试的堆叠生成对抗网络鲁棒性评估

Robustness Evaluation of Stacked Generative Adversarial Networks using Metamorphic Testing

Monash University Malaysia(莫纳什大学马来西亚分校)

Hyejin Park, Taaha Waseem, Wen Qi Teo, Ying Hwei Low, Mei Kuan Lim, Chun Yong Chong

摘要:

从自然语言中合成具有真实感的图像是计算机视觉的一个挑战性问题。在过去的十年中,已经提出了许多方法,其中改进的堆栈生成对抗网络(StackGAN-v2)已被证明能够生成反映输入文本描述中指定细节的高分辨率图像。在本文中,我们的目标是通过引入训练数据的变化来评估StackGAN-v2模型的鲁棒性和容错能力。但由于生成对抗网络(GAN)的工作原理,当训练数据被修改时,很难预测模型的输出。因此,在这项工作中,我们采用变形测试技术来评估模型的鲁棒性与各种意想不到的训练数据集。因此,我们首先实现StackGAN-v2算法,并测试原作者提供的预训练模型,为我们的实验建立基础事实。然后,我们确定一个变形关系,从测试用例生成。根据试验结果的观察,推导出了变形关系式。最后,我们综合了所有变形关系的实验结果,发现StackGAN-v2算法容易受到带有突出对象的输入图像的影响,即使它与主要对象的重叠最小,这是StackGAN-v2模型的作者和用户没有报道的。所提出的变形关系可以应用到其他文本到图像的合成模型,不仅验证了鲁棒性,而且还帮助研究人员理解和解释的结果由机器学习模型。

英文摘要:

Synthesising photo-realistic images from natural language is one of the challenging problems in computer vision. Over the past decade, a number of approaches have been proposed, of which the improved Stacked Generative Adversarial Network (StackGAN-v2) has proven capable of generating high resolution images that reflect the details specified in the input text descriptions. In this paper, we aim to assess the robustness and fault-tolerance capability of the StackGAN-v2 model by introducing variations in the training data. However, due to the working principle of Generative Adversarial Network (GAN), it is difficult to predict the output of the model when the training data are modified. Hence, in this work, we adopt Metamorphic Testing technique to evaluate the robustness of the model with a variety of unexpected training dataset. As such, we first implement StackGAN-v2 algorithm and test the pre-trained model provided by the original authors to establish a ground truth for our experiments. We then identify a metamorphic relation, from which test cases are generated. Further, metamorphic relations were derived successively based on the observations of prior test results. Finally, we synthesise the results from our experiment of all the metamorphic relations and found that StackGAN-v2 algorithm is susceptible to input images with obtrusive objects, even if it overlaps with the main object minimally, which was not reported by the authors and users of StackGAN-v2 model. The proposed metamorphic relations can be applied to other text-to-image synthesis models to not only verify the robustness but also to help researchers understand and interpret the results made by the machine learning models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑