arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.03158cs.LGcs.DScs.ITcs.NImath.IT
收藏

最优与近似最优自适应矢量量化

Optimal and Near-Optimal Adaptive Vector Quantization

UCL(伦敦大学学院);VMware Research(VMware研究);Harvard University(哈佛大学)

Ran Ben-Basat, Yaniv Ben-Itzhak, Michael Mitzenmacher, Shay Vargaftik

AI总结:

针对最优自适应量化方法运行与内存开销高、不可行的问题,该研究重新探究自适应矢量量化问题,提出了复杂度渐近优化的最优算法及面向大输入的快速近似最优算法,为其在更多机器学习场景的应用铺路。

摘要:

量化是众多机器学习应用场景中的一项基础优化手段,应用场景包括压缩梯度、模型权重、激活值以及数据集。精度最高的量化形式是自适应量化,即针对给定输入最小化误差,而非针对最坏情况进行优化。然而,最优自适应量化方法在运行时间和内存需求方面都被认为不具备可行性。\n我们重新研究了自适应矢量量化(Adaptive Vector Quantization,AVQ)问题,提出了能够找到最优解的算法,其时间和空间复杂度在渐近意义上得到了提升。我们还针对大规模输入提出了一种速度更快的近似最优算法。我们的实验表明,所提算法有望推动AVQ在各类机器学习应用中得到更广泛的应用。

英文摘要:

Quantization is a fundamental optimization for many machine-learning use cases, including compressing gradients, model weights and activations, and datasets. The most accurate form of quantization is \emph{adaptive}, where the error is minimized with respect to a given input, rather than optimizing for the worst case. However, optimal adaptive quantization methods are considered infeasible in terms of both their runtime and memory requirements. We revisit the Adaptive Vector Quantization (AVQ) problem and present algorithms that find optimal solutions with asymptotically improved time and space complexity. We also present an even faster near-optimal algorithm for large inputs. Our experiments show our algorithms may open the door to using AVQ more extensively in a variety of machine learning applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑