Deep neural networks (DNNs) have become one of the enabling technologies in many safety-critical applications, e.g., autonomous driving and medical image analysis. DNN systems, however, suffer from various kinds of threats, such as adversarial example attacks and fault injection attacks. While there are many defense methods proposed against maliciously crafted inputs, solutions against faults presented in the DNN system itself (e.g., parameters and calculations) are far less explored. In this paper, we develop a novel lightweight fault-tolerant solution for DNN-based systems, namely DeepDyve, which employs pre-trained neural networks that are far simpler and smaller than the original DNN for dynamic verification. The key to enabling such lightweight checking is that the smaller neural network only needs to produce approximate results for the initial task without sacrificing fault coverage much. We develop efficient and effective architecture and task exploration techniques to achieve optimized risk/overhead trade-off in DeepDyve. Experimental results show that DeepDyve can reduce 90% of the risks at around 10% overhead.
Deep learning has transformed the way we think of software and what it can do. But deep neural networks are fragile and their behaviors are often surprising. In many settings, we need to provide formal guarantees on the safety, security, correctness, or robustness of neural networks. This book covers foundational ideas from formal verification and their adaptation to reasoning about neural networks and deep learning.
Although neural networks are widely used, it remains challenging to formally verify the safety and robustness of neural networks in real-world applications. Existing methods are designed to verify the network before deployment, which are limited to relatively simple specifications and fixed networks. These methods are not ready to be applied to real-world problems with complex and/or dynamically changing specifications and networks. To effectively handle such problems, verification needs to be performed online when these changes take place. However, it is still challenging to run existing veri…
Shanghai Jiao Tong University（上海交通大学）；WeBank（微众银行）；Hong Kong University of Science and Technology（香港科技大学）
Federated learning models are collaboratively developed upon valuable training data owned by multiple parties. During the development and deployment of federated models, they are exposed to risks including illegal copying, re-distribution, misuse and/or free-riding. To address these risks, the ownership verification of federated learning models is a prerequisite that protects federated learning model intellectual property rights (IPR) i.e., FedIPR. We propose a novel federated deep neural network (FedDNN) ownership verification scheme that allows private watermarks to be embedded and verified …
Adversarial-Aware Deep Learning System based on a Secondary Classical Machine Learning Verification Approach
College of Engineering and Computer Science, University of Central Florida（中佛罗里达大学工程与计算机科学学院）；College of Computer Engineering and Science, Prince Sattam bin Abdulaziz University（沙特亲王萨勒曼·本·阿卜杜勒阿齐兹大学计算机工程与科学学院）；College of Engineering, SUNY Polytechnic Institute（纽约州立大学理工学院工程学院）
Automatic Road Subsurface Distress Recognition from Ground Penetrating Radar Images using Deep Learning-based Cross-verification
Department of Engineering Mechanics, School of Civil Engineering and Transportation, South China University of Technology（华南理工大学土木工程与交通学院力学系）；Guangdong Provincial Academy of Building Research Group Co. Ltd.（广东省建筑科学研究集团有限公司）；School of Civil and Transportation Engineering, Guangdong University of Technology（广东工业大学土木与交通学院）
Improving Neural Network Verification through Spurious Region Guided Refinement
Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Intelligent Software（智能软件研究所）；Zhejiang University（浙江大学）；Singapore Management University（新加坡管理大学）
Generating Synthetic Satellite Imagery With Deep-Learning Text-to-Image Models -- Technical Challenges and Implications for Monitoring and Verification
Berliner Hochschule für Technik（柏林应用技术大学）；Princeton University（普林斯顿大学）；Einstein Center Digital Future（爱因斯坦数字未来中心）
Max Planck Institute for Software Systems（马克斯·普朗克软件系统研究所）；TU Darmstadt（达姆施塔特工业大学）
Formal verification has emerged as a powerful approach to ensure the safety and reliability of deep neural networks. However, current verification tools are limited to only a handful of properties that can be expressed as first-order constraints over the inputs and output of a network. While adversarial robustness and fairness fall under this category, many real-world properties (e.g., "an autonomous vehicle has to stop in front of a stop sign") remain outside the scope of existing verification technology. To mitigate this severe practical restriction, we introduce a novel framework for verify…
Verification Code Recognition Based on Active and Deep Learning
Shandong University（山东大学）；Harbin Institute of Technology（哈尔滨工业大学）；Temple University（天普大学）；Xidian University（西安电子科技大学）；North China Electric Power University（华北电力大学）；Capital Normal University（首都师范大学）；Intermediate People's Court（中级人民法院）
The use of neural networks as function approximators has enabled many advances in reinforcement learning (RL). The generalization power of neural networks combined with advances in RL algorithms has reignited the field of artificial intelligence. Despite their power, neural networks are considered black boxes, and their use in safety-critical settings remains a challenge. Recently, neural network verification has emerged as a way to certify safety properties of networks. Verification is a hard problem, and it is difficult to scale to large networks such as the ones used in deep reinforcement l…
Partial AUC optimization based deep speaker embeddings with class-center learning for text-independent speaker verification
Northwestern Polytechnical University（西北工业大学）
基于深度嵌入的文本无关说话人确认在许多具有挑战性的场景中表现出优于传统方法的性能。其损失函数一般可分为两类，即，核查和识别。验证损失函数与说话人验证的流水线相匹配，但其实现比较困难。因此，大多数最先进的深度嵌入方法使用具有softmax输出单元或其变体的识别损失函数。在本文中，我们提出了一个验证损失函数，命名为最大化的接收机操作特征（ROC）曲线下的部分面积（pAUC），基于深度嵌入文本无关的说话人验证。我们还提出了一种基于类中心的训练试验构建方法，以提高训练效率，这是关键的建议损失函数是可比的识别损失的性能。在Speaker in the Wild（SITW）和NIST SRE 2016数据集上的实验表明，所提出的pAUC损失函数与最先进的识别损失函数具有很强的竞争力。
A Survey of Safety and Trustworthiness of Deep Neural Networks: Verification, Testing, Adversarial Attack and Defence, and Interpretability
University of Liverpool（利物浦大学）；University of Oxford（牛津大学）；Lancaster University（兰卡斯特大学）；Defence Science and Technology Laboratory (Dstl)（国防科学技术实验室）；Queen’s University Belfast（贝尔法斯特女王大学）
Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models
Carnegie Mellon University（卡内基梅隆大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Salesforce Research（赛富时研究院）；Harvard Medical School, Harvard University（哈佛医学院（哈佛大学））；Fanhan Tech.（泛函科技）
East China Normal University（华东师范大学）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）
Occlusion is a prevalent and easily realizable semantic perturbation to deep neural networks (DNNs). It can fool a DNN into misclassifying an input image by occluding some segments, possibly resulting in severe errors. Therefore, DNNs planted in safety-critical systems should be verified to be robust against occlusions prior to deployment. However, most existing robustness verification approaches for DNNs are focused on non-semantic perturbations and are not suited to the occlusion case. In this paper, we propose the first efficient, SMT-based approach for formally verifying the occlusion robu…
Deep neural networks (DNNs) have gained significant popularity in recent years, becoming the state of the art in a variety of domains. In particular, deep reinforcement learning (DRL) has recently been employed to train DNNs that realize control policies for various types of real-world systems. In this work, we present the whiRL 2.0 tool, which implements a new approach for verifying complex properties of interest for DRL systems. To demonstrate the benefits of whiRL 2.0, we apply it to case studies from the communication networks domain that have recently been used to motivate formal verifica…
CAPM: Fast and Robust Verification on Maxpool-based CNN via Dual Network
Graduate Institute of Communication Engineering, National Taiwan University（国立台湾大学通信工程研究所）；Department of Electrical Engineering, National Taiwan University（国立台湾大学电子工程系）；MediaTek Research（联发科技研究）
National University of Singapore（新加坡国立大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；The University of Queensland（昆士兰大学）；Griffith University（格里菲斯大学）
Neural networks have been widely applied in security applications such as spam and phishing detection, intrusion prevention, and malware detection. This black-box method, however, often has uncertainty and poor explainability in applications. Furthermore, neural networks themselves are often vulnerable to adversarial attacks. For those reasons, there is a high demand for trustworthy and rigorous methods to verify the robustness of neural network models. Adversarial robustness, which concerns the reliability of a neural network when dealing with maliciously manipulated inputs, is one of the hot…
Deep speaker embedding extractors have already become new state-of-the-art systems in the speaker verification field. However, the problem of verification score calibration for such systems often remains out of focus. An irrelevant score calibration leads to serious issues, especially in the case of unknown acoustic conditions, even if we use a strong speaker verification system in terms of threshold-free metrics. This paper presents an investigation over several methods of score calibration: a classical approach based on the logistic regression model; the recently presented magnitude estimati…
University of Pennsylvania（宾夕法尼亚大学）；Massachusetts Institute of Technology（麻省理工学院）；Carnegie Mellon University（卡内基梅隆大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Analyzing the worst-case performance of deep neural networks against input perturbations amounts to solving a large-scale non-convex optimization problem, for which several past works have proposed convex relaxations as a promising alternative. However, even for reasonably-sized neural networks, these relaxations are not tractable, and so must be replaced by even weaker relaxations in practice. In this work, we propose a novel operator splitting method that can directly solve a convex relaxation of the problem to high accuracy, by splitting it into smaller sub-problems that often have analytic…
Koopman Regularized Deep Speech Disentanglement for Speaker Verification
School of Electronics and Computer Science, University of Southampton（电子与计算机科学学院，南安普顿大学）；Marine Geosciences, National Oceanography Centre（海洋科学，国家海洋学中心）
