Synthesising photo-realistic images from natural language is one of the challenging problems in computer vision. Over the past decade, a number of approaches have been proposed, of which the improved Stacked Generative Adversarial Network (StackGAN-v2) has proven capable of generating high resolution images that reflect the details specified in the input text descriptions. In this paper, we aim to assess the robustness and fault-tolerance capability of the StackGAN-v2 model by introducing variations in the training data. However, due to the working principle of Generative Adversarial Network (GAN), it is difficult to predict the output of the model when the training data are modified. Hence, in this work, we adopt Metamorphic Testing technique to evaluate the robustness of the model with a variety of unexpected training dataset. As such, we first implement StackGAN-v2 algorithm and test the pre-trained model provided by the original authors to establish a ground truth for our experiments. We then identify a metamorphic relation, from which test cases are generated. Further, metamorphic relations were derived successively based on the observations of prior test results. Finally, we synthesise the results from our experiment of all the metamorphic relations and found that StackGAN-v2 algorithm is susceptible to input images with obtrusive objects, even if it overlaps with the main object minimally, which was not reported by the authors and users of StackGAN-v2 model. The proposed metamorphic relations can be applied to other text-to-image synthesis models to not only verify the robustness but also to help researchers understand and interpret the results made by the machine learning models.
Longitudinal Prediction of Postnatal Brain Magnetic Resonance Images via a Metamorphic Generative Adversarial Network
Nanjing University of Information Science and Technology(南京信息工程大学);Radboud University Medical Center(拉德堡德大学医学中心);Shandong University (Weihai)(山东大学(威海))
VITAL: A Visual Interpretation on Text with Adversarial Learning for Image Labeling
School of Computer Science, Wuhan University, Wuhan, Hubei, China(中国湖北省武汉市武汉大学计算机学院);School of Information Engineering, Hubei University for Nationalities, Enshi, Hubei, China(中国湖北省恩施市湖北民族学院信息工程学院)
DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis
Nanjing University of Posts and Telecommunications(南京邮电大学);ETH Zürich(苏黎世联邦理工学院);Wuhan University(武汉大学);Peng Cheng Laboratory(鹏城实验室);University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, CAS(中国科学院自动化研究所)
Applying Conditional Generative Adversarial Networks for Imaging Diagnosis
University of Houston(休斯顿大学);Johns Hopkins University(约翰斯·霍普金斯大学);Northeastern University(东北大学);University of Massachusetts Boston(马萨诸塞大学波士顿分校);Brandeis University(布兰迪斯大学)
Metamorphic Testing for Audio Content Moderation Software
School of Information, Renmin University of China(中国人民大学信息学院);Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程系);Department of Electrical Engineering, City University of Hong Kong(香港城市大学电子工程系);Department of Computer Science and Engineering, Hong Kong University of Science and Technology(香港科学大学计算机科学与工程系)
Learning to Find Missing Video Frames with Synthetic Data Augmentation: A General Framework and Application in Generating Thermal Images Using RGB Cameras
Laboratory for Intelligent & Safe Automobiles (LISA)(智能与安全汽车实验室(LISA));University of California San Diego(加利福尼亚大学圣迭戈分校);Visual Analysis and Perception Lab(视觉分析与感知实验室);Aalborg Universitet(奥尔堡大学)
Test It Before You Trust It: Applying Software Testing for Trustworthy In-context Learning
Advanced Institute of So-Go-Chi (Convergence Knowledge) Informatics(融合知识研究院);Tohoku University(东北大学);Japan Advanced Institute of Science and Technology(日本先进科学研究院);Faculty of Information and Communication Technology(信息与通信技术学院);Mahidol University(玛希敦大学)