Colby Banbury, Vijay Janapa Reddi, Peter Torelli, Jeremy Holleman, Nat Jeffries, Csaba Kiraly, Pietro Montino, David Kanter, Sebastian Ahmed, Danilo Pau, Urmish… 展开作者
Colby Banbury, Vijay Janapa Reddi, Peter Torelli, Jeremy Holleman, Nat Jeffries, Csaba Kiraly, Pietro Montino, David Kanter, Sebastian Ahmed, Danilo Pau, Urmish Thakker, Antonio Torrini, Peter Warden, Jay Cordaro, Giuseppe Di Guglielmo, Javier Duarte, Stephen Gibellini, Videet Parekh, Honson Tran, Nhan Tran, Niu Wenxu, Xu Xuesong
Advancements in ultra-low-power tiny machine learning (TinyML) systems promise to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted and easily reproducible benchmark for these systems. To meet this need, we present MLPerf Tiny, the first industry-standard benchmark suite for ultra-low-power tiny machine learning systems. The benchmark suite is the collaborative effort of more than 50 organizations from industry and academia and reflects the needs of the community. MLPerf Tiny measures the accuracy, latency, and energy of machine learning inference to properly evaluate the tradeoffs between systems. Additionally, MLPerf Tiny implements a modular design that enables benchmark submitters to show the benefits of their product, regardless of where it falls on the ML deployment stack, in a fair and reproducible manner. The suite features four benchmarks: keyword spotting, visual wake words, image classification, and anomaly detection.
Facebook（脸书）；University of Colorado, Boulder（科罗拉多大学博尔德分校）；Google Research（谷歌研究院）；University of Minnesota（明尼苏达大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；Netflix（奈飞）
Deep learning-based recommendation models are used pervasively and broadly, for example, to recommend movies, products, or other information most relevant to users, in order to enhance the user experience. Among various application domains which have received significant industry and academia research attention, such as image classification, object detection, language and speech translation, the performance of deep learning-based recommendation models is less well explored, even though recommendation tasks unarguably represent significant AI inference cycles at large-scale datacenter fleets. T…
Heidelberg University（海德堡大学）；Columbia University（哥伦比亚大学）；University of California San Diego（加州大学圣地亚哥分校）；European Organization for Nuclear Research (CERN)（欧洲核子研究组织（CERN））；Fermi National Accelerator Laboratory（费米国家加速器实验室）；University of Washington（华盛顿大学）；Harvard University（哈佛大学）；AMD Adaptive and Embedded Computing Group (AECG) Labs（AMD自适应与嵌入式计算集团（AECG）实验室）
We present our development experience and recent results for the MLPerf Tiny Inference Benchmark on field-programmable gate array (FPGA) platforms. We use the open-source hls4ml and FINN workflows, which aim to democratize AI-hardware codesign of optimized neural networks on FPGAs. We present the design and implementation process for the keyword spotting, anomaly detection, and image classification benchmark tasks. The resulting hardware implementations are quantized, configurable, spatial dataflow architectures tailored for speed and efficiency and introduce new generic optimizations and comm…
Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；Argonne National Laboratory（阿贡国家实验室）；Maxwell Labs（麦克斯韦实验室）；Swiss National Supercomputing Centre（瑞士国家超级计算中心）；RIKEN Center for Computational Science（日本理化学研究所计算科学研究中心）；University of Virginia（弗吉尼亚大学）；MLCommons；NVIDIA（英伟达）；Google（谷歌）；National Center for Supercomputing Applications（美国国家超级计算应用中心）；Texas Advanced Computing Center（德克萨斯高级计算中心）；Fujitsu Limited（富士通株式会社）；Oak Ridge National Laboratory（橡树岭国家实验室）；Microsoft（微软）；Cruise；National Institute of Advanced Industrial Science and Technology（日本产业技术综合研究所）
Scientific communities are increasingly adopting machine learning and deep learning models in their applications to accelerate scientific insights. High performance computing systems are pushing the frontiers of performance with a rich diversity of hardware resources and massive scale-out capabilities. There is a critical need to understand fair and effective benchmarking of machine learning applications that are representative of real-world scientific use cases. MLPerf is a community-driven standard to benchmark machine learning workloads, focusing on end-to-end performance metrics. In this p…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）；Dell Inc（戴尔公司）
MLPerf, an emerging machine learning benchmark suite strives to cover a broad range of applications of machine learning. We present a study on its characteristics and how the MLPerf benchmarks differ from some of the previous deep learning benchmarks like DAWNBench and DeepBench. We find that application benchmarks such as MLPerf (although rich in kernels) exhibit different features compared to kernel benchmarks such as DeepBench. MLPerf benchmark suite contains a diverse set of models which allows unveiling various bottlenecks in the system. Based on our findings, dedicated low latency interc…
University of California, Davis（加州大学戴维斯分校）；Arm（ARM公司）；Samsung（三星）；Qualcomm（高通）；California State University, Sacramento（加州州立大学萨克拉门托分校）；Gilmet Labs（Gilmet实验室）；NVIDIA（英伟达）；AMD（超微半导体）
The recent submission of Google TPU-v3 Pods to the industry wide MLPerf v0.6 training benchmark demonstrates the scalability of a suite of industry relevant ML models. MLPerf defines a suite of models, datasets and rules to follow when benchmarking to ensure results are comparable across hardware, frameworks and companies. Using this suite of models, we discuss the optimizations and techniques including choice of optimizer, spatial partitioning and weight update sharding necessary to scale to 1024 TPU chips. Furthermore, we identify properties of models that make scaling them challenging, such…
Google（谷歌）；Intel（英特尔）；Stanford University（斯坦福大学）；Landing AI；NVIDIA（英伟达）；University of California, Berkeley（加州大学伯克利分校）；Harvard University（哈佛大学）；Cisco（思科）；Facebook（脸书）；Cerebras；Real World Technologies；Synopsys（新思科技）；University of Toronto（多伦多大学）；Vector Institute；Tesla（特斯拉）；Alibaba（阿里巴巴）
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can increase the time to solution, training is stochastic and time to solution exhibits high variance, and software and hardware systems are so diverse that fair benchmarking with the same binary, code, and even hyperparameters is difficult. We therefore present MLPerf, an ML benchmark th…
This paper presents the first industry-standard open-source machine learning (ML) benchmark to allow perfor mance and accuracy evaluation of mobile devices with different AI chips and software stacks. The benchmark draws from the expertise of leading mobile-SoC vendors, ML-framework providers, and model producers. It comprises a suite of models that operate with standard data sets, quality metrics and run rules. We describe the design and implementation of this domain-specific ML benchmark. The current benchmark version comes as a mobile app for different computer vision and natural language p…
Deep learning for recommendation data is one of the most pervasive and challenging AI workload in recent times. State-of-the-art recommendation models are one of the largest models matching the likes of GPT-3 and Switch Transformer. Challenges in deep learning recommendation models (DLRM) stem from learning dense embeddings for each of the categorical tokens. These embedding tables in industrial scale models can be as large as hundreds of terabytes. Such large models lead to a plethora of engineering challenges, not to mention prohibitive communication overheads, and slower training and infere…
Harvard University（哈佛大学）；Intel（英特尔）；Real World Insights；Google（谷歌）；Microsoft（微软）；Facebook（脸书）；NVIDIA（英伟达）；Qualcomm（高通）；Stanford University（斯坦福大学）；Myrtle；Landing AI；Mythic；Advantage Engineering；Habana Labs（Habana实验室）；Alibaba T-Head（阿里巴巴平头哥）；MediaTek（联发科）；OPPO；Synopsys（新思科技）；dividiti；Arm；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Xilinx（赛灵思）；Tesla（特斯拉）；Alibaba（阿里巴巴）；Centaur Technology（半人马技术公司）；Alibaba Cloud（阿里云）；General Motors（通用汽车）；Tencent（腾讯）；Biren Research（壁仞研究院）
Machine-learning (ML) hardware and software system demand is burgeoning. Driven by ML applications, the number of different ML inference systems has exploded. Over 100 organizations are building ML inference chips, and the systems that incorporate existing models span at least three orders of magnitude in power consumption and five orders of magnitude in performance; they range from embedded devices to data-center solutions. Fueling the hardware are a dozen or more software frameworks and libraries. The myriad combinations of ML hardware and ML software make assessing ML-system performance in …
Harvard University（哈佛大学）；Samsung Semiconductor, Inc.（三星半导体公司）；Syntiant（Syntiant公司）；University of North Carolina, Charlotte（北卡罗来纳大学夏洛特分校）；Cisco Systems（思科系统公司）；California State Polytechnic University, Pomona（加州州立理工大学波莫纳分校）；Real World Insights（Real World Insights公司）；dividiti（dividiti公司）；University of California, Berkeley（加州大学伯克利分校）；Google（谷歌）；STMicroelectronics, Italy（意法半导体（意大利））；Arizona State University（亚利桑那州立大学）；Reality AI（Reality AI公司）；Arm ML Research Lab（Arm机器学习研究实验室）；KU Leuven（鲁汶大学）；Interuniversity Microelectronics Centre (IMEC)（校际微电子中心（IMEC））；University of York（约克大学）
Recent advancements in ultra-low-power machine learning (TinyML) hardware promises to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted benchmark for these systems. Benchmarking allows us to measure and thereby systematically compare, evaluate, and improve the performance of systems and is therefore fundamental to a field reaching maturity. In this position paper, we present the current landscape of TinyML and discuss the challenges and direction towards developing a fair and useful hardware benchmark for TinyML workload…
Recent advances in state-of-the-art ultra-low power embedded devices for machine learning (ML) have permitted a new class of products whose key features enable ML capabilities on microcontrollers with less than 1 mW power consumption (TinyML). TinyML provides a unique solution by aggregating and analyzing data at the edge on low-power embedded devices. However, we have only recently been able to run ML on microcontrollers, and the field is still in its infancy, which means that hardware, software, and research are changing extremely rapidly. Consequently, many TinyML frameworks have been devel…
University of Westminster（威斯敏斯特大学）；University of New South Wales（新南威尔士大学）
Tiny Machine Learning (TinyML) is an upsurging research field that proposes to democratize the use of Machine Learning and Deep Learning on highly energy-efficient frugal Microcontroller Units. Considering the general assumption that TinyML can only run inference, growing interest in the domain has led to work that makes them reformable, i.e., solutions that permit models to improve once deployed. This work presents a survey on reformable TinyML solutions with the proposal of a novel taxonomy. Here, the suitability of each hierarchical layer for reformability is discussed. Furthermore, we expl…
Peng Cheng Laboratory（鹏城实验室）；Tsinghua University（清华大学）；Institute of Computing Technology Chinese Academy of Sciences（中国科学院计算技术研究所）
The plethora of complex artificial intelligence (AI) algorithms and available high performance computing (HPC) power stimulates the expeditious development of AI components with heterogeneous designs. Consequently, the need for cross-stack performance benchmarking of AI-HPC systems emerges rapidly. The de facto HPC benchmark LINPACK can not reflect AI computing power and I/O performance without representative workload. The current popular AI benchmarks like MLPerf have fixed problem size therefore limited scalability. To address these issues, we propose an end-to-end benchmark suite utilizing …
Qualcomm AI Research（高通人工智能研究院）；Qualcomm Canada ULC（高通加拿大无限责任公司）；University of Toronto（多伦多大学）
Tiny machine learning (tinyML) has emerged during the past few years aiming to deploy machine learning models to embedded AI processors with highly constrained memory and computation capacity. Low precision quantization is an important model compression technique that can greatly reduce both memory consumption and computation cost of model inference. In this study, we focus on post-training quantization (PTQ) algorithms that quantize a model to low-bit (less than 8-bit) precision with only a small set of calibration data and benchmark them on different tinyML use cases. To achieve a fair compa…
Researchers have proposed hardware, software, and algorithmic optimizations to improve the computational performance of deep learning. While some of these optimizations perform the same operations faster (e.g., increasing GPU clock speed), many others modify the semantics of the training procedure (e.g., reduced precision), and can impact the final model's accuracy on unseen data. Due to a lack of standard evaluation criteria that considers these trade-offs, it is difficult to directly compare these optimizations. To address this problem, we recently introduced DAWNBench, a benchmark competiti…
Siemens AG（西门子股份公司）；Technical University of Munich（慕尼黑工业大学）
Tiny machine learning (TinyML) has gained widespread popularity where machine learning (ML) is democratized on ubiquitous microcontrollers, processing sensor data everywhere in real-time. To manage TinyML in the industry, where mass deployment happens, we consider the hardware and software constraints, ranging from available onboard sensors and memory size to ML-model architectures and runtime platforms. However, Internet of Things (IoT) devices are typically tailored to specific tasks and are subject to heterogeneity and limited resources. Moreover, TinyML models have been developed with diff…
Tiny Machine Learning (TML) is a new research area whose goal is to design machine and deep learning techniques able to operate in Embedded Systems and IoT units, hence satisfying the severe technological constraints on memory, computation, and energy characterizing these pervasive devices. Interestingly, the related literature mainly focused on reducing the computational and memory demand of the inference phase of machine and deep learning models. At the same time, the training is typically assumed to be carried out in Cloud or edge computing systems (due to the larger memory and computationa…
University of Alberta（阿尔伯塔大学）；IIT Kharagpur（印度理工学院卡拉格普尔分校）
The focus of this paper is a proof of concept, machine learning (ML) pipeline that extracts heart rate from pressure sensor data acquired on low-power edge devices. The ML pipeline consists an upsampler neural network, a signal quality classifier, and a 1D-convolutional neural network optimized for efficient and accurate heart rate estimation. The models were designed so the pipeline was less than 40 kB. Further, a hybrid pipeline consisting of the upsampler and classifier, followed by a peak detection algorithm was developed. The pipelines were deployed on ESP32 edge device and benchmarked ag…
