沿输入维度剖析 FLOPs 以实现 GreenAI 成本估算
Dissecting FLOPs along input dimensions for GreenAI cost estimations
AI总结:
本文提出 α-FLOPs 指标,通过沿输入维度修正卷积层浮点运算次数计算公式,使 GreenAI 成本估算更贴近并行硬件上的实际能耗。
摘要:
GreenAI 这一术语指的是一种新颖的深度学习方法,它更加关注其方法的生态影响和计算效率。GreenAI 的倡导者建议使用浮点运算次数(FLOPs)作为神经网络计算成本的度量;然而,该度量与配备 GPU 或 TPU 等大规模并行处理单元的硬件能耗相关性并不好。在本文中,我们对用于计算卷积层浮点运算次数的公式提出了一种简单的改进,称为 α-FLOPs,解释并修正了传统上相对于不同层的偏差,使其更接近现实。α-FLOPs 的概念基于一个关键洞察:在输入具有多个维度的情况下,没有理由相信并行化所提供的加速会沿着所有不同轴均匀分布。
英文摘要:
The term GreenAI refers to a novel approach to Deep Learning, that is more aware of the ecological impact and the computational efficiency of its methods. The promoters of GreenAI suggested the use of Floating Point Operations (FLOPs) as a measure of the computational cost of Neural Networks; however, that measure does not correlate well with the energy consumption of hardware equipped with massively parallel processing units like GPUs or TPUs. In this article, we propose a simple refinement of the formula used to compute floating point operations for convolutional layers, called α-FLOPs, explaining and correcting the traditional discrepancy with respect to different layers, and closer to reality. The notion of α-FLOPs relies on the crucial insight that, in case of inputs with multiple dimensions, there is no reason to believe that the speedup offered by parallelism will be uniform along all different axes.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。