通过贝尔盒量化提升熵
Boosting Entropy with Bell Box Quantization
AI总结：
BBQ是一种信息最优且计算高效的量化方法，通过在输入领域执行信息最优量化并在输出领域映射到计算高效类型，显著提升了模型的困惑度性能。
摘要：
量化感知预训练（QAPT）是一种有效的技术，用于减少深度神经网络的计算和内存开销，同时提高其在边缘设备上的能效。现有QAPT方法生成的模型存储在计算高效的数据显示类型（例如整数）中，这些类型在信息理论上并不最优（ITO）。另一方面，现有的ITO数据类型（例如分位数/正态浮点量化）并不计算高效。我们提出了BBQ，这是首个同时具有信息最优性和计算高效性的量化方法。BBQ基于我们的关键见解，即由于学习是领域无关的，量化器的输出不需要与其输入处于同一领域。BBQ在其输入领域中执行信息最优的量化，并将其输出转换到一个计算高效的领域，其中信息最优的数据类型映射到计算高效的数据类型。在不牺牲计算效率的情况下，BBQ在4位模型中将困惑度减少多达2点，在3位模型中减少多达4点，在2位模型中减少多达5点，在1位模型中减少多达18点。代码可在https://github.com/1733116199/bbq上获得。
英文摘要：
Quantization-Aware Pre-Training (QAPT) is an effective technique to reduce the compute and memory overhead of Deep Neural Networks while improving their energy efficiency on edge devices. Existing QAPT methods produce models stored in compute-efficient data types (e.g. integers) that are not information theoretically optimal (ITO). On the other hand, existing ITO data types (e.g. Quantile/NormalFloat Quantization) are not compute-efficient. We propose BBQ, the first ITO quantization method that is also compute-efficient. BBQ builds on our key insight that since learning is domain-agnostic, the output of a quantizer does not need to reside in the same domain as its input. BBQ performs ITO quantization in its input domain, and returns its output in a compute-efficient domain where ITO data types are mapped to compute-efficient data types. Without sacrificing compute efficiency, BBQ outperforms prior SOTA QAPT methods by a perplexity reduction of up to 2 points for 4-bit models, up to 4 points for 3-bit models, up to 5 points for 2-bit models, and up to 18 points for 1-bit models. Code is available at https://github.com/1733116199/bbq.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
