向量量化中度量普适性的代价至多为0.11比特
Price of metric universality in vector quantization is at most 0.11 bit
AI总结:
本文证明存在一个通用码本,对于所有可能的X统计量,在W为高斯时,其性能至少与速率每维度降低0.11比特的X自适应水填充码本相当。
摘要:
快速计算矩阵乘积 $W^\top X$ 是现代大语言模型的核心操作。为了更高效地部署,一种流行的方法是使用低精度近似 $\widehat W$ 替代真实 $W$(“仅权重量化”)。信息论表明,降低 $W$ 精度的最优算法依赖于 $X$ 的(二阶)统计量,并且需要将向量量化码本与 $X$ 的 PCA 方向仔细对齐(称为“水填充分配”的过程)。然而,码本对 $X$ 统计量的依赖性非常不实用。本文证明存在一个通用码本,对于所有可能的 $X$ 统计量同时接近最优,其意义在于:当 $W$ 为高斯时,该通用码本至少与速率每维度降低 0.11 比特的 $X$ 自适应水填充码本一样好。这样的通用码本将是低精度存储格式的理想候选者,这是当前活跃研究的话题,但可惜存在性证明是非构造性的。等价地,我们的结果表明在 $\mathbb{R}^n$ 中存在一个网,它同时关于所有希尔伯特范数是球面的接近最优覆盖。
英文摘要:
Fast computation of a matrix product $W^\top X$ is a workhorse of modern LLMs. To make their deployment more efficient, a popular approach is that of using a low-precision approximation $\widehat W$ in place of true $W$ (``weight-only quantization''). Information theory demonstrates that an optimal algorithm for reducing precision of $W$ depends on the (second order) statistics of $X$ and requires a careful alignment of vector quantization codebook with PCA directions of $X$ (a process known as ``waterfilling allocation''). Dependence of the codebook on statistics of $X$, however, is highly impractical. This paper proves that there exist a universal codebook that is simultaneously near-optimal for all possible statistics of $X$, in the sense of being at least as good as an $X$-adapted waterfilling codebook with rate reduced by 0.11 bit per dimension in the case when $W$ is Gaussian. Such universal codebook would be an ideal candidate for the low-precision storage format, a topic of active modern research, but alas the existence proof is non-constructive. Equivalently, our result shows existence of a net in $\mathbb{R}^n$ that is a nearly-optimal covering of a sphere simultaneously with respect to all Hilbert norms.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。