压缩还是不压缩？利用指数浓度推动生成式AI模型权重无损压缩的前沿
To Compress or Not? Pushing the Frontier of Lossless GenAI Model Weights Compression with Exponent Concentration
AI总结：
该研究发现生成式AI权重存在指数浓度的统计规律，推导了接近FP4.67的理论压缩极限，提出结合熵感知编码与GPU优化解码的ECF8无损压缩框架，在最高671B参数模型上实现显著内存节省与吞吐量提升且计算完全无损。
摘要：
生成式AI（GenAI）模型规模扩展至数千亿参数，使得低精度计算成为高效部署不可或缺的技术。我们认为根本解决方案在于开发低精度浮点格式，这类格式天然具备数值稳定性、内存节省和硬件效率优势，且无反量化开销。本文对GenAI权重中的指数浓度现象展开理论与实证研究：跨架构与模态下，权重指数始终呈现低熵特性。我们证明这一现象由随机梯度下降诱导的α-稳定分布自然产生，并推导了指数熵的紧致界。我们的分析确立了接近FP4.67的理论压缩极限，这为实用FP8格式的设计提供了依据。基于这些洞见，我们提出指数集中型FP8（ECF8）——一种结合熵感知编码与GPU优化解码的无损压缩框架。在参数规模最高达671B的大语言模型（LLMs）和扩散Transformer（DiTs）上的实验表明，该框架可实现最高26.9%的内存节省和177.1%的吞吐量提升，且计算完全无损，即模型输出无任何偏差。我们的研究结果将指数浓度确立为训练后模型的一条统计规律，并为FP8时代的无损低精度浮点设计开辟了一条有理论依据的路径。
英文摘要：
The scaling of Generative AI (GenAI) models into the hundreds of billions of parameters makes low-precision computation indispensable for efficient deployment. We argue that the fundamental solution lies in developing low-precision floating-point formats, which inherently provide numerical stability, memory savings, and hardware efficiency without dequantization overhead. In this paper, we present a theoretical and empirical study of an exponent concentration phenomenon in GenAI weights: exponents consistently exhibit low entropy across architectures and modalities. We show that this arises naturally from $α$-stable distributions induced by stochastic gradient descent, and we prove tight bounds on the entropy of exponents. Our analysis establishes a theoretical compression limit near FP4.67, which motivates the design of a practical FP8 format. Building on these insights, we propose Exponent-Concentrated FP8 (ECF8), a lossless compression framework with entropy-aware encoding and GPU-optimized decoding. Experiments on LLMs and DiTs up to 671B parameters demonstrate up to 26.9% memory savings and 177.1% throughput acceleration, with perfectly lossless computations, i.e., no deviation in model outputs. Our results establish exponent concentration as a statistical law of trained models and open a principled path for lossless low-precision floating-point design in the FP8 era.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
