分布内一致性正则化提升量化感知训练的泛化能力
In-Distribution Consistency Regularization Improves the Generalization of Quantization-Aware Training
AI总结：
针对量化感知训练（QAT）性能下降的问题，提出利用分布内一致性正则化同时解决训练不稳定与测试泛化差的问题，经理论与实验验证可有效提升性能，在CIFAR-10上的ResNet18与MobileNet上均有显著提升，甚至优于全精度模型。
摘要：
尽管现有量化感知训练（Quantization-Aware Training，QAT）方法高度依赖知识蒸馏来保证性能，但QAT仍存在严重的性能下降问题。实验表明，普通量化对来自输入和权重的扰动都很敏感。因此我们假设，QAT的泛化能力问题主要由内在不稳定性（训练阶段）和有限的泛化能力（测试阶段）共同导致。本文从新视角出发，利用一致性正则化（Consistency Regular，CR）同时解决这两个问题，以提升QAT的泛化能力。实证结果与理论分析验证了CR能为不同网络架构、各类QAT方法带来良好的泛化能力。大量实验表明，该方法显著优于当前最先进的QAT方法，甚至优于全精度（FP）对照模型。在CIFAR-10数据集上，使用ResNet18时该方法相比基线提升3.79%，使用轻量模型MobileNet时相比基线提升3.84%。
英文摘要：
Although existing Quantization-Aware Training (QAT) methods intensively depend on knowledge distillation to guarantee performance, QAT still suffers from severe performance drop. The experiments have shown that vanilla quantization is sensitive to the perturbation from both the input and weights. Therefore, we assume that the generalization ability of QAT is predominantly caused by both the intrinsic instability (training time) and the limited generalization ability (testing time). In this paper, we address both issues from a new perspective by leveraging Consistency Regularization (CR) to improve the generalization ability of QAT. Empirical results and theoretical analysis verify that CR would bring a good generalization ability to different network architectures and various QAT methods. Extensive experiments demonstrate that our approach significantly outperforms current state-of-the-art QAT methods and even the FP counterparts. On CIFAR-10, the proposed method improves by 3.79% compared to the baseline method using ResNet18, and improves by 3.84% compared to the baseline method using the lightweight model MobileNet.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
