This paper proposes a hardware-oriented dropout algorithm, which is efficient for field programmable gate array (FPGA) implementation. In deep neural networks (DNNs), overfitting occurs when networks are overtrained and adapt too well to training data. Consequently, they fail in predicting unseen data used as test data. Dropout is a common technique that is often applied in DNNs to overcome this problem. In general, implementing such training algorithms of DNNs in embedded systems is difficult due to power and memory constraints. Training DNNs is power-, time-, and memory- intensive; however, embedded systems require low power consumption and real-time processing. An FPGA is suitable for embedded systems for its parallel processing characteristic and low operating power; however, due to its limited memory and different architecture, it is difficult to apply general neural network algorithms. Therefore, we propose a hardware-oriented dropout algorithm that can effectively utilize the characteristics of an FPGA with less memory required. Software program verification demonstrates that the performance of the proposed method is identical to that of conventional dropout, and hardware synthesis demonstrates that it results in significant resource reduction.
Enhancing Dropout-based Bayesian Neural Networks with Multi-Exit on FPGA
Imperial College London(帝国理工学院);University College London(伦敦大学学院);University of Sydney(悉尼大学);Hunan Normal University(湖南师范大学);Samsung AI Center(三星人工智能中心);University of Cambridge(剑桥大学)
Spatial-SpinDrop: Spatial Dropout-based Binary Bayesian Neural Network with Spintronics Implementation
Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);Univ. Grenoble Alpes(格勒诺布尔阿尔卑斯大学);CEA(法国原子能和替代能源委员会);CNRS(法国国家科学研究中心);Grenoble INP(格勒诺布尔国立理工学院);IRIG-Spintec(格勒诺布尔微电子与纳米技术研究所自旋电子与纳米技术实验室);RevoAI GmbH(RevoAI有限公司)
Scale-Dropout: Estimating Uncertainty in Deep Neural Networks Using Stochastic Scale
Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);Univ. Grenoble Alpes(格勒诺布尔阿尔卑斯大学);CEA(法国原子能和替代能源委员会);CNRS(法国国家科学研究中心);Grenoble INP(格勒诺布尔国立理工学院);IRIG-Spintec(格勒诺布尔微电子与纳米技术研究所-自旋电子学与纳米技术实验室);RevoAI GmbH(RevoAI有限公司)
Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers
Northeastern University(东北大学);Simon Fraser University(西蒙菲莎大学);ETH Zurich(苏黎世联邦理工学院);University of Connecticut(康涅狄格大学);Beijing University of Technology(北京工业大学)
Hardware-aware Training Techniques for Improving Robustness of Ex-Situ Neural Network Transfer onto Passive TiO2 ReRAM Crossbars
IT (Institut Interdisciplinaire d’Innovation Technologique)(跨学科技术创新研究所);Université de Sherbrooke(舍布鲁克大学);Institute of Electronics, Microelectronics and Nanotechnology (IEMN), Université de Lille(里尔大学电子、微电子与纳米技术研究所)
Tensor computing application programming interface: An application-oriented, lightweight API for portable high-performance tensor network computations
Department of Physics and Astronomy, California State University Northridge(加州州立大学北岭分校物理与天文学系);RIKEN Interdisciplinary Theoretical and Mathematical Sciences Program (iTHEMS)(理化学研究所交叉学科理论与数学科学项目);Computational Materials Science Research Team, RIKEN Center for Computational Science (R-CCS)(理化学研究所计算科学中心计算材料科学研究团队);Quantum Computational Science Research Team, RIKEN Center for Quantum Computing (RQC)(理化学研究所量子计算中心量子计算科学研究团队);Computational Condensed Matter Physics Laboratory, RIKEN Cluster for Pioneering Research (CPR)(理化学研究所先锋研究集群计算凝聚态物理实验室);Quantum Computing Simulation Unit, Quantum-HPC Hybrid Platform Division, RIKEN Center for Computational Science (R-CCS)(理化学研究所计算科学中心量子计算模拟单元);Next-Generation HPC Application Development Unit, Next-Generation HPC Infrastructure Development Division, RIKEN Center for Computational Science (R-CCS)(理化学研究所计算科学中心下一代HPC应用开发单元)