Statistical models are widely used to estimate the performance of commercial off-the-shelf (COTS) AI hardware accelerators. However, training of statistical performance models often requires vast amounts of data, leading to a significant time investment and can be difficult in case of limited hardware availability. To alleviate this problem, we propose a novel performance modeling methodology that significantly reduces the number of training samples while maintaining good accuracy. Our approach leverages knowledge of the target hardware architecture and initial parameter sweeps to identify a set of Performance Representatives (PR) for deep neural network (DNN) layers. These PRs are then used for benchmarking, building a statistical performance model, and making estimations. This targeted approach drastically reduces the number of training samples needed, opposed to random sampling, to achieve a better estimation accuracy. We achieve a Mean Absolute Percentage Error (MAPE) of as low as 0.02% for single-layer estimations and 0.68% for whole DNN estimations with less than 10000 training samples. The results demonstrate the superiority of our method for single-layer estimations compared to models trained with randomly sampled datasets of the same size.
Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approaches
Applied AI(应用人工智能);Artificial Intelligence Research Institute(人工智能研究所);HSE University(俄罗斯高等经济大学);Sb AI Lab(Sb人工智能实验室);MWS AI(MWS人工智能);Yandex;Independent Researcher(独立研究者)
KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware
Department of Computer Science and Technology, University of Cambridge, Cambridge, United Kingdom(计算机科学与技术系,剑桥大学,剑桥,英国);Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom(电气与电子工程系,伦敦帝国理工学院,伦敦,英国);School of Informatics, University of Edinburgh, Edinburgh, United Kingdom(信息学院,爱丁堡大学,爱丁堡,英国)
Benchmarking TinyML Systems: Challenges and Direction
Harvard University(哈佛大学);Samsung Semiconductor, Inc.(三星半导体公司);Syntiant(Syntiant公司);University of North Carolina, Charlotte(北卡罗来纳大学夏洛特分校);Cisco Systems(思科系统公司);California State Polytechnic University, Pomona(加州州立理工大学波莫纳分校);Real World Insights(Real World Insights公司);dividiti(dividiti公司);University of California, Berkeley(加州大学伯克利分校);Google(谷歌);STMicroelectronics, Italy(意法半导体(意大利));Arizona State University(亚利桑那州立大学);Reality AI(Reality AI公司);Arm ML Research Lab(Arm机器学习研究实验室);KU Leuven(鲁汶大学);Interuniversity Microelectronics Centre (IMEC)(校际微电子中心(IMEC));University of York(约克大学)
DABench-LLM: Standardized and In-Depth Benchmarking of Post-Moore Dataflow AI Accelerators for LLMs
Dept. of Computer Science Stevens Institute of Technology(计算机科学系 斯坦福理工学院);Dept. of Computer Science Binghamton University(计算机科学系 哈伯里顿大学);Engineering The Ohio State University(工程学 俄亥俄州立大学);Learning Division Argonne National Laboratory(学习部 阿贡国家实验室)
Benchmarking Machine Learning Models for IoT Malware Detection under Data Scarcity and Drift
Department of Computer Science, The College of Wooster, Wooster, USA(沃斯特学院计算机科学系);Department of Computer Science and Engineering, University of Nevada-Reno, Reno, USA(内华达大学雷诺分校计算机科学与工程系)
Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers
Kaze Technologies(凯兹技术);Kaze Consulting(凯兹咨询);Electrical and Computer Engineering(电气与计算机工程);Lebanese American University(黎巴嫩美国大学);Carnegie Mellon University(卡内基梅隆大学);University of Exeter(埃克塞特大学)