Ensuring solution feasibility is a key challenge in developing Deep Neural Network (DNN) schemes for solving constrained optimization problems, due to inherent DNN prediction errors. In this paper, we propose a ``preventive learning'' framework to guarantee DNN solution feasibility for problems with convex constraints and general objective functions without post-processing, upon satisfying a mild condition on constraint calibration. Without loss of generality, we focus on problems with only inequality constraints. We systematically calibrate inequality constraints used in DNN training, thereby anticipating prediction errors and ensuring the resulting solutions remain feasible. We characterize the calibration magnitudes and the DNN size sufficient for ensuring universal feasibility. We propose a new Adversarial-Sample Aware training algorithm to improve DNN's optimality performance without sacrificing feasibility guarantee. Overall, the framework provides two DNNs. The first one from characterizing the sufficient DNN size can guarantee universal feasibility while the other from the proposed training algorithm further improves optimality and maintains DNN's universal feasibility simultaneously. We apply the framework to develop DeepOPF+ for solving essential DC optimal power flow problems in grid operation. Simulation results over IEEE test cases show that it outperforms existing strong DNN baselines in ensuring 100% feasibility and attaining consistent optimality loss ($<$0.19%) and speedup (up to $\times$228) in both light-load and heavy-load regimes, as compared to a state-of-the-art solver. We also apply our framework to a non-convex problem and show its performance advantage over existing schemes.
Bayesian Hyperparameter Optimization for Deep Neural Network-Based Network Intrusion Detection
Kennesaw State University(肯尼索州立大学);Florida International University(佛罗里达国际大学);United International University(联合国际大学);University of Calabria(卡拉布里亚大学);LORIA(洛林计算机科学及其应用实验室);Tuskegee University(塔斯基吉大学)
Robust DNN Partitioning and Resource Allocation Under Uncertain Inference Time
Beijing National Research Center for Information Science and Technology, Department of Electronic Engineering, Tsinghua University(北京信息科学与技术国家研究中心,电子工程系,清华大学)
Progressive DNN Compression: A Key to Achieve Ultra-High Weight Pruning and Quantization Rates using ADMM
Sensetime Research(商汤科技研究院);Tsinghua University(清华大学);Syracuse University(雪城大学);Northeastern University(东北大学);Florida International University(佛罗里达国际大学);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);IBM Research(IBM研究院)
权值修剪和权值量化是DNN模型压缩的两个重要方面。这些技术的先前工作主要基于化学。最近的一项工作使用先进的优化技术ADMM(Alternating Direction Methods of Multipliers)开发了DNN权重修剪的系统框架,实现了最先进的权重修剪结果之一。在这项工作中,我们首先扩展这样的一杆ADMM为基础的框架,以保证解决方案的可行性,并提供快速收敛速度,以及推广到权重量化。我们进一步开发了一个多步骤的渐进式DNN权重修剪和量化框架,具有双重好处:(i)由于ADMM正则化的特殊属性,实现了进一步的权重修剪/量化,以及(ii)减少了每个步骤中的搜索空间。大量的实验结果表明,与以前的工作相比,优越的性能。一些亮点:(i)我们分别在LeNet-5、AlexNet和ResNet-50模型上实现了246倍、36倍和8倍的权重修剪,(几乎)零精度损失;(ii)在AlexNet中甚至实现了显著的61倍权重修剪(ImageNet)与先前的工作相比,实际准确性仅略有下降;(iii)我们是第一批为ResNet和MobileNet模型得出显着权重修剪结果的人之一;(iv)我们为MNIST和CIFAR-10分别导出了第一个无损的、完全二进制化的(所有层)LeNet-5和VGG-16;以及(v)我们为ImageNet导出了第一个完全二进制化的(所有层)ResNet,具有合理的准确性损失…
Ecomap: Sustainability-Driven Optimization of Multi-Tenant DNN Execution on Edge Servers
School of Electrical, Computer and Biomedical Engineering, Southern Illinois University(南伊利诺伊大学电气、计算机与生物医学工程学院);The University of Texas at Austin(德克萨斯大学奥斯汀分校)
NeMo: A Neuron-Level Modularizing-While-Training Approach for Decomposing DNN Models
State Key Laboratory of Complex \& Critical Software Environment, Beihang University Beijing China;Peng Cheng Laboratory Shenzhen China;National University of Singapore Singapore;Beihang University Beijing China;Hangzhou Innovation Institute of Beihang University Hangzhou China;State Key Laboratory of Complex \& Critical Software Environment, Beihang University;Peng Cheng Laboratory;National University of Singapore;Beihang University;Hangzhou Innovation Institute of Beihang University
AdapSNE: Adaptive Fireworks-Optimized and Entropy-Guided Dataset Sampling for Edge DNN Training
School of Software Engineering(软件工程学院);School of Computer Science and Technology(计算机科学与技术学院);the First Affiliated Hospital of Xi’an Jiaotong University(西安交通大学第一附属医院);National Key Laboratory of Human-Machine Hybrid Augmented Intelligence(人机混合增强智能国家重点实验室);National Engineering Research Center for Visual Information and Applications(视觉信息与应用国家工程研究中心);Institute of Artificial Intelligence and Robotics(人工智能与机器人研究院)