解析Transformer：一种面向绿色AI的CLEAR视角
Dissecting Transformers: A CLEAR Perspective towards Green AI
AI总结：
本文提出CLEAR方法，通过组件级能耗评估，分析Transformer组件在不同参数下的能耗，揭示注意力机制的高能耗问题，为预测能效提供理论基础。
摘要：
大规模语言模型的快速应用引发了显著的环境问题。与训练成本不同，LLM推理持续发生，主导了AI能耗足迹。然而，大多数可持续性研究仅报告粗略的模型级指标，将能效视为次要目标。为解决这一限制，我们提出了通过重复进行组件级能耗评估的CLEAR方法，以克服微秒级组件执行与毫秒级能耗传感器监控之间的时间不匹配。使用CLEAR，我们评估了15个模型，涵盖四种架构类型，在组件级能耗波动低于9.5%的同时，捕捉了超过90%的总能耗作为独立组件。我们展示了Transformer组件在关键参数如批次大小、注意力头数、隐藏维度、KV缓存和注意力变体上的首次全面、细致的能耗分析。我们的发现表明，与整个模型相比，注意力机制每FLOP消耗的能耗显著更高，表明FLOPs单独无法捕捉真实的组件级能耗成本。CLEAR使可靠细致的能耗测量成为可能，并为能耗消耗的预测建模提供了坚实的理论基础。
英文摘要：
The rapid adoption of Large Language Models (LLMs) has raised significant environmental concerns. Unlike the one-time cost of training, LLM inference occurs continuously and dominates the AI energy footprint. Yet most sustainability studies report only coarse model-level metrics, treating energy efficiency as an afterthought rather than a primary objective. Addressing the limitation, we propose Component-Level Energy Assessment via Repetitions CLEAR, to overcome temporal mismatch between microsecond scale component execution and millisecond(ms) scale monitoring of energy sensors. Using CLEAR, we evaluate 15 models spanning four architecture types, keeping component-wise energy variance below 9.5% while capturing over 90% of total energy as individual components. We present the first comprehensive, fine-grained energy analysis of Transformer components across key parameters such as batch size, attention heads, hidden dimension, KV cache, and attention variants. Our findings reveal that Attention consumes significantly more Energy per FLOP as compared to the entire model, indicating that FLOPs alone fail to capture true component-level energy cost. CLEAR enables reliable fine-grained energy measurements and provides a strong formal foundation for predictive modelling of energy consumption.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
