The power that machine learning models consume when making predictions can be affected by a model's architecture. This paper presents various estimates of power consumption for a range of different activation functions, a core factor in neural network model architecture design. Substantial differences in hardware performance exist between activation functions. This difference informs how power consumption in machine learning models can be reduced.
On Output Activation Functions for Adversarial Losses: A Theoretical Analysis via Variational Divergence Minimization and An Empirical Study on MNIST Classification
University of California San Diego（加州大学圣地亚哥分校）；Academia Sinica（中央研究院）；Taiwan AI Labs（台湾人工智能实验室）
Variations on the Chebyshev-Lagrange Activation Function
Winterlight Labs（温特莱特实验室）；University of Toronto（多伦多大学）；Surgical Safety Technologies（手术安全技术公司）；Li Ka Shing Knowledge Institute, St Michael’s Hospital（圣迈克尔医院李嘉诚知识研究所）；Vector Institute for Artificial Intelligence（矢量人工智能研究所）
Department of Physics and Center for Theoretical Physics, National Taiwan University（物理系与理论物理中心，国立台湾大学）；Center for Quantum Science and Engineering, National Taiwan University（量子科学与工程中心，国立台湾大学）；Physics Division, National Center for Theoretical Sciences, National Taiwan University（理论科学国家中心物理组，国立台湾大学）；Department of Computer Science, University of North Carolina at Chapel Hill（计算机科学系，北卡罗来纳大学教堂山分校）
Neural networks for neurocomputing circuits: a computational study of tolerance to noise and activation function non-uniformity when machine learning materials properties
National Taiwan University（国立台湾大学）；Institute of Science Tokyo（东京科学大学）
CauchyNet: Compact and Data-Efficient Learning using Holomorphic Activation Functions
School of Sciences, Great Bay University, Dongguan 52300, China（广东东莞大湾大学科学学院）；Department of Mathematics and Statistics, University of Massachusetts, Amherst（马萨诸塞大学阿姆赫斯特分校数学与统计学系）；Great Bay Institute for Advanced Study, Great Bay University, China（大湾大学高级研究所）；School of Computing and Information Technology, Great Bay University, Dongguan 52300, China（广东东莞大湾大学计算与信息科技学院）；Department of Mathematics, Northwestern University, Evanston, IL, USA（西北大学数学系）
Integral Signatures of Activation Functions: A 9-Dimensional Taxonomy and Stability Theory for Deep Learning
Bellini College of AI, Cybersecurity and Computing（人工智能、网络安全与计算学院）；University of South Florida（佛罗里达州立大学）；College of Computing and Informatics（计算与信息学院）；Drexel University（德雷塞尔大学）；Departments of Physics（物理系）
Accelerating Large Language Model Training with 4D Parallelism and Memory Consumption Estimator
School of Computing, Institute of Science Tokyo（科学技术学院计算机学院）；Turing. inc（图灵公司）；Super Computing Research Center, Institute of Science Tokyo（科学技术学院超级计算研究中心）
