Optimal transport (OT) is a popular tool in machine learning to compare probability measures geometrically, but it comes with substantial computational burden. Linear programming algorithms for computing OT distances scale cubically in the size of the input, making OT impractical in the large-sample regime. We introduce a practical algorithm, which relies on a quantization step, to estimate OT distances between measures given cheap sample access. We also provide a variant of our algorithm to improve the performance of approximate solvers, focusing on those for entropy-regularized transport. We give theoretical guarantees on the benefits of this quantization step and display experiments showing that it behaves well in practice, providing a practical approximation algorithm that can be used as a drop-in replacement for existing OT estimators.
LSQ+: Improving low-bit quantization through learnable offsets and better initialization
Qualcomm AI Research, Qualcomm Technologies, Inc.(高通AI研究,高通技术公司);Qualcomm AI Research, Qualcomm Technologies Netherlands B.V.(高通AI研究,高通技术荷兰有限公司);Seoul National University(首尔大学)
Improving Quantization-aware Training of Low-Precision Network via Block Replacement on Full-Precision Counterpart
College of Information Science and Electronic Engineering, Zhejiang University(浙江大学信息科学与电子工程学院);ZJU-UIUC Institute, Zhejiang University(浙江大学-伊利诺伊大学厄巴纳-香槟分校联合研究院)
Improving the Robustness of Quantized Deep Neural Networks to White-Box Attacks using Stochastic Quantization and Information-Theoretic Ensemble Training
Center for Vision Technologies(视觉技术中心);SRI International(SRI国际)
Precision Gating: Improving Neural Network Efficiency with Dynamic Dual-Precision Activations
Cornell University(康奈尔大学);Cornell Tech(康奈尔科技学院)
我们提出了精确门控(PG),这是一种用于深度神经网络的端到端可训练动态双精度量化技术。PG以低精度计算大多数特征,仅以较高精度计算一小部分重要特征以保持准确性。所提出的方法适用于各种DNN架构,并显着降低了DNN执行的计算成本,几乎没有精度损失。我们的实验表明,PG在CNN上取得了优异的结果,包括静态压缩的移动友好网络,如ShuffleNet。与最先进的基于预测的量化方案相比,PG在ImageNet上实现了相同或更高的精度,但计算量减少了2.4 $\times$。PG也适用于RNN。与8位均匀量化相比,PG在Penn Tree Bank数据集上的LSTM上每字的困惑度提高了1.2%,计算成本降低了2.7 $\times$。代码可在:https://github.com/cornell-zhang/dnn-gating
RaanA: A Fast, Flexible, and Data-Efficient Post-Training Quantization Algorithm
Computer Science and Engineering, University of Michigan(密歇根大学计算机科学与工程系);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算机科学与数据科学学院);Physics of Artificial Intelligence Group, NTT Research, Inc.(NTT研究所人工智能物理组)