使用随机哈达玛变换进行量化：一种高效的启发式方法已被证明
Quantizing With Randomized Hadamard Transforms: Efficient Heuristic Now Proven
AI总结：
本文研究了随机哈达玛变换在量化中的应用，证明了使用两个变换可近似均匀随机旋转的效果，并展示了三个变换在向量量化中的优势。
摘要：
均匀随机旋转（URRs）是现代量化方法中常用的预处理步骤，用于梯度压缩、推理加速、KV-cache压缩、模型权重量化和向量数据库中的近似最近邻搜索。在实践中，URRs常被随机哈达玛变换（RHTs）替代，后者在保持正交性的同时允许快速实现。剩余的问题是对于最坏情况输入的性能。使用URRs时，每个坐标独立分布为移位的Beta分布，这在高维情况下收敛于高斯分布。通常，一个RHT在最坏情况下并不合适，因为单个坐标可能远离这些分布。我们证明，对任意大小为d的输入向量应用两次RHT后，归一化旋转向量的每个固定坐标的边缘分布与标准高斯分布的Kolmogorov距离和1-Wasserstein距离均在O(d^{-1/2})范围内。然后我们将这些界限应用于现代压缩方案，即DRIVE和QUIC-FL，并证明两次RHT在渐近上可与URRs性能匹配。然而，我们证明两次RHT可能不足以用于向量量化（VQ），因为VQ通常需要固定大小块坐标之间的弱相关性（而不是仅单个坐标的边缘分布收敛）。我们证明三个RHT的组合导致坐标协方差衰减。这确保了任何固定、有界、多维VQ代码本，若优化用于URRs，则在使用三个RHT时，其预期误差相同，仅在加法项上与维度成反比。最后，由于实际输入很少是对抗性的，我们提出了一种线性时间O(d)的检查，用于动态调整运行时使用的RHT数量以提高性能。
英文摘要：
Uniform random rotations (URRs) are a common preprocessing step in modern quantization approaches used for gradient compression, inference acceleration, KV-cache compression, model weight quantization, and approximate nearest-neighbor search in vector databases. In practice, URRs are often replaced by randomized Hadamard transforms (RHTs), which preserve orthogonality while admitting fast implementations. The remaining issue is the performance for worst-case inputs. With a URR, each coordinate is distributed as a shifted beta distribution, which converges to a Gaussian distribution in high dimensions. Generally, one RHT is not suitable in the worst case, as individual coordinates can be far from these distributions. We show that after composing two RHTs on any $d$-sized input vector, the marginal distribution of every fixed coordinate of the normalized rotated vector is within $O(d^{-1/2})$ of a standard Gaussian both in Kolmogorov distance and in $1$-Wasserstein distance. We then plug these bounds into the analyses of modern compression schemes, namely DRIVE and QUIC-FL, and show that two RHTs achieve performance that asymptotically matches URRs. However, while two RHTs suffice for scalar quantization, they may be insufficient for Vector Quantization (VQ), which depends on the joint distribution across fixed-size blocks of coordinates (as opposed to only marginal distribution convergence for single coordinates). We prove that a composition of three RHTs leads to decaying coordinate covariance. This ensures that any fixed, bounded, multi-dimensional VQ codebook optimized for URRs has the same expected error when using three RHTs, up to an additive term that vanishes with the dimension. Finally, because practical inputs are rarely adversarial, we propose a linear-time $O(d)$ check on the input's moments to dynamically adapt the number of RHTs used at runtime to improve performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
