划分引导的K-Means：面向极端模型压缩的极端空簇解决方法
Partitioning-Guided K-Means: Extreme Empty Cluster Resolution for Extreme Model Compression
AI总结：
提出PG k-means方法，通过划分预分配、启发式解决和优化步骤解决空簇问题，在GLUE基准上平均减少100倍空簇，提升模型准确率最高达12%。
摘要：
深度学习中的紧凑性对于模型在低资源应用中的可行性至关重要，而极端模型压缩的常见方法是量化。我们认为带有Quant-Noise的迭代乘积量化（iPQ）是该领域的最先进技术，但该量化框架由于普遍存在的空簇而遭受可预防的推理质量下降。在本文中，我们提出了几种新颖的增强方法，旨在通过重点关注解决空簇来提高带有Quant-Noise的iPQ的准确性。我们的贡献被称为划分引导的k-means（PG k-means），这是一个大幅增强的k-means实现，由三个主要组件组成。首先，我们提出了一种基于划分的预分配策略，确保没有初始空簇并鼓励权重到簇的均匀分布。其次，我们提出了一种经验上更优的空簇解决启发式方法，通过对大簇的谨慎划分来执行。最后，我们构建了一个可选的优化步骤，合并直观上密集的权重簇以确保共享表示。所提出的方法在应用于GLUE基准测试中各种任务的RoBERTa时，平均将带有Quant-Noise的iPQ中的空簇数量减少了100倍，在空簇解决过程中使用的迭代次数减少了8倍，并将整体模型准确率提高了多达12%。
英文摘要：
Compactness in deep learning can be critical to a model's viability in low-resource applications, and a common approach to extreme model compression is quantization. We consider Iterative Product Quantization (iPQ) with Quant-Noise to be state-of-the-art in this area, but this quantization framework suffers from preventable inference quality degradation due to prevalent empty clusters. In this paper, we propose several novel enhancements aiming to improve the accuracy of iPQ with Quant-Noise by focusing on resolving empty clusters. Our contribution, which we call Partitioning-Guided k-means (PG k-means), is a heavily augmented k-means implementation composed of three main components. First, we propose a partitioning-based pre-assignment strategy that ensures no initial empty clusters and encourages an even weight-to-cluster distribution. Second, we propose an empirically superior empty cluster resolution heuristic executed via cautious partitioning of large clusters. Finally, we construct an optional optimization step that consolidates intuitively dense clusters of weights to ensure shared representation. The proposed approach consistently reduces the number of empty clusters in iPQ with Quant-Noise by 100x on average, uses 8x fewer iterations during empty cluster resolution, and improves overall model accuracy by up to 12%, when applied to RoBERTa on a variety of tasks in the GLUE benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
