DPQuant: 通过动态量化调度实现高效且差分隐私的模型训练
DPQuant: Efficient and Differentially-Private Model Training via Dynamic Quantization Scheduling
AI总结：
本文提出DPQuant框架，通过动态选择量化层和损失感知优先级，减少量化方差，提升差分隐私训练的精度与效率，实现在低精度硬件上2.21倍的吞吐量提升。
摘要：
差分隐私随机梯度下降（DP-SGD）及其自适应变种DP-Adam是保护用户隐私的强大技术，当使用敏感数据训练神经网络时。在训练过程中，将模型权重和激活转换为低精度格式，即量化，可以大幅减少训练时间、能耗和成本，因此被广泛使用。在本工作中，我们首次证明量化在DP训练中导致的精度退化显著高于常规SGD。我们发现这是由噪声注入引起的，放大了量化方差，导致不成比例的精度退化。为了解决这一挑战，我们提出了DPQuant，一个动态量化框架，能够在每个epoch动态选择一个变化的层子集进行量化。我们的方法结合了两个关键思想，有效减少量化方差：（i）概率采样，每epoch旋转哪些层被量化，以及（ii）损失感知的层优先级，使用差分隐私损失敏感度估计器来识别可以量化而对模型质量影响最小的层。该估计器消耗了总体隐私预算的微小比例，保持DP保证。在ResNet18、ResNet50和DenseNet121在多种数据集上的实证评估表明，DPQuant在静态量化基线上表现一致，实现了接近帕累托最优的精度-计算量权衡，并在低精度硬件上实现了高达2.21倍的理论吞吐量提升，验证精度下降不到2%。我们进一步显示，我们的框架可扩展到DP-Adam，获得相似的收益。
英文摘要：
Differentially-Private SGD (DP-SGD) and its adaptive variant DP-Adam are powerful techniques to protect user privacy when using sensitive data to train neural networks. During training, converting model weights and activations into low-precision formats, i.e., quantization, can drastically reduce training times, energy consumption, and cost, and is thus a widely used technique. In this work, we demonstrate for the first time that quantization causes significantly higher accuracy degradation in DP training compared to regular SGD. We observe that this is caused by noise injection, which amplifies quantization variance, leading to disproportionately large accuracy degradation. To address this challenge, we present DPQuant, a dynamic quantization framework that adaptively selects a changing subset of layers to quantize at each epoch. Our method combines two key ideas that effectively reduce quantization variance: (i) probabilistic sampling that rotates which layers are quantized every epoch, and (ii) loss-aware layer prioritization, which uses a differentially private loss sensitivity estimator to identify layers that can be quantized with minimal impact on model quality. This estimator consumes a negligible fraction of the overall privacy budget, preserving DP guarantees. Empirical evaluations on ResNet18, ResNet50, and DenseNet121 across a range of datasets demonstrate that DPQuant consistently outperforms static quantization baselines, achieving near Pareto-optimal accuracy-compute trade-offs and up to $2.21\times$ theoretical throughput improvements on low-precision hardware, with less than 2% drop in validation accuracy. We further show that our framework extends to DP-Adam with similar gains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
