RATQ：一个通用的定长随机优化量化器
RATQ: A Universal Fixed-Length Quantizer for Stochastic Optimization
摘要：
我们提出了旋转自适应四迭代量化器（RATQ），一个固定长度的量化器的梯度在一阶随机优化。RATQ易于实现，仅涉及Hadamard变换计算和具有适当选择的动态范围的自适应均匀量化。噪声梯度几乎肯定有界欧氏范数，我们建立了一个信息理论的下限优化精度使用有限精度梯度，并表明RATQ几乎达到这个下限。对于均方有界的噪声梯度，我们使用增益形状量化器，它分别量化的欧几里德范数和使用RATQ的归一化单位范数向量。我们建立下限的任何优化程序和形状量化器的性能，当使用一个均匀的增益量化器。最后，我们提出了一个自适应量化器的增益，当使用RATQ形状量化优于均匀增益量化，事实上，接近最佳。作为一个副产品，我们表明，我们的固定长度量化器RATQ具有几乎相同的性能作为最佳的可变长度量化器的分布均值估计。此外，我们得到了一个有效的量化器的高斯向量，达到一个率非常接近的高斯率失真函数，事实上，是普遍的亚高斯输入向量。
英文摘要：
We present Rotated Adaptive Tetra-iterated Quantizer (RATQ), a fixed-length quantizer for gradients in first order stochastic optimization. RATQ is easy to implement and involves only a Hadamard transform computation and adaptive uniform quantization with appropriately chosen dynamic ranges. For noisy gradients with almost surely bounded Euclidean norms, we establish an information theoretic lower bound for optimization accuracy using finite precision gradients and show that RATQ almost attains this lower bound. For mean square bounded noisy gradients, we use a gain-shape quantizer which separately quantizes the Euclidean norm and uses RATQ to quantize the normalized unit norm vector. We establish lower bounds for performance of any optimization procedure and shape quantizer, when used with a uniform gain quantizer. Finally, we propose an adaptive quantizer for gain which when used with RATQ for shape quantizer outperforms uniform gain quantization and is, in fact, close to optimal. As a by-product, we show that our fixed-length quantizer RATQ has almost the same performance as the optimal variable-length quantizers for distributed mean estimation. Also, we obtain an efficient quantizer for Gaussian vectors which attains a rate very close to the Gaussian rate-distortion function and is, in fact, universal for subgaussian input vectors.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
