FLOPs与实际工作量：AI效率评估中复现的重要性

中文摘要 AI 辅助

由于模型规模庞大、能源需求高以及环境成本高，AI效率近来在学术界和工业界都受到了关注。虽然报告浮点运算量（FLOPs）是评估计算成本的传统方法，但FLOPs与执行时间之间的关系并不直接，因为相同FLOPs数量的层可能具有不同的执行时间，原因是某些操作更容易并行化。本文旨在复现一项提出了α-FLOPs估算公式的研究的原始实验，以验证其结果在更新、更强大的硬件上是否仍然适用。在复现过程中，我们发现原始研究提供的复现材料存在局限性，包括缺乏具体的依赖细节以及回归数据的透明度不足。我们的结果验证了这一论点：仅原始FLOPs并不是评估执行时间的合适指标，因为空间维度比核维度更容易并行化。然而，细粒度测量表明，这种关系比之前显示的要复杂得多，新硬件在执行时间上表现出不稳定性和不连续性，包括跳变和振荡，而α-FLOPs公式通常会低估这些情况。最终，本工作验证了原始研究的实证发现，但在应用α-FLOPs估算时得到了负面结果。我们还强调了针对依赖硬件的效率评估研究，拥有完整准确的复现包的迫切需求，并提供了我们实现的完整复现包以促进进一步研究。

英文摘要

AI efficiency has recently taken the spotlight in both academy and industry due to massive model scales, high energy demands, and environmental costs. While reporting Floating Point Operations (FLOPs) is a traditional approach for assessing computational costs, the relationship between FLOPs and execution time is not straightforward, as layers with the same number of FLOPs may not have the same execution time because some operations are more easily parallelized than others. This paper sets out to replicate the original experiments from a study that proposed the $α-FLOPs$ estimation formula to verify whether the results remain applicable on newer, more powerful hardware. During the replication process, we identify limitations in the replication materials provided by the original study, including a lack of specific dependency details and transparency regarding regression data. Our results validate the thesis that raw FLOPs alone are not an appropriate metric for execution time, as spatial dimensions remain more easily parallelized than kernel dimensions. However, fine-grained measurements reveal that the relationship is much less straightforward than previously shown, with newer hardware exhibiting instabilities and discontinuities in execution time, including jumps and oscillations, that the $α-FLOPs$ formula generally underestimates. Ultimately, this work validates the empirical findings from the original study but shows negative results when applying the $α-FLOPs$ estimation. We also highlight the critical need for complete and accurate replication packages for research on hardware-dependent efficiency assessment and provide a complete replication package for our implementation to facilitate further study.

发表机构

• Delft University of Technology（代尔夫特理工大学）

机构由 AI 辅助整理，请以论文原文为准。