arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.09696cs.LGcs.AI
收藏

消失贡献:一种用于平滑和迭代模型压缩的统一框架

Vanishing Contributions: A Unified Framework for Smooth and Iterative Model Compression

Department of Electronic and Telecommunication, Politecnico di Torino(电子与电信系,托斯纳大学);King Abdullah University of Science and Technology(卡布斯科学与技术大学);Department of Electrical, Electronic, and Information Engineering, University of Bologna(电气、电子与信息工程系,博洛尼亚大学);Advanced Research Center on Electronic Systems (ARCES), University of Bologna(电子系统高级研究中心(ARCES),博洛尼亚大学)

Lorenzo Nikiforos, Luciano Prono, Charalampos Antoniadis, Fabio Pareschi, Riccardo Rovatti, Gianluca Setti

AI总结:

本文提出VCON框架,通过并行执行原始和压缩模型,逐步降低原始模型贡献,提升压缩模型贡献,从而提高稳定性和准确性。

摘要:

随着深度神经网络规模的扩大,需要压缩技术如剪枝、量化和低秩分解。尽管这些方法有效减少内存、计算和能耗,但可能引入严重精度下降,通常通过迭代渐进压缩缓解。然而,不同压缩技术需要不同的迭代方法,且某些导致不稳定、断续的模型微调。我们引入消失贡献(VCON),一种统一框架,实现DNN平滑迭代压缩。VCON在微调期间同时执行原始和压缩模型,逐步降低原始模型贡献,同时增加压缩模型贡献。这种仿射组合使网络缓慢适应,提高稳定性和缓解精度下降。我们在计算机视觉和自然语言处理基准上评估VCON,使用多种压缩策略。在大多数设置中,框架优于后剪枝和迭代基线。典型增益超过1%,某些配置改进超过15%。VCON因此兼容现有压缩技术,并在多样化任务中持续提升性能。

英文摘要:

The increasing scale of Deep Neural Networks (DNNs) introduces the need for compression techniques such as pruning, quantization, and low-rank decomposition. While these methods are very effective at reducing memory, computation, and energy consumption, they may introduce severe accuracy degradation, which is often mitigated by using iterative, gradual compression. However, different compression techniques require distinct iterative approaches, and some result in unstable, discontinuous model fine-tuning. We introduce Vanishing Contributions (VCON), a unified framework for the smooth, iterative transition of DNNs into a compressed form. Rather than replacing the original network directly with its compressed version, VCON executes both in parallel during fine-tuning. The contribution of the original (uncompressed) model is progressively reduced, while that of the compressed model is gradually increased. This affine combination allows the network to slowly adapt, improving stability and mitigating accuracy degradation. We evaluate VCON on computer vision and natural language processing benchmarks, using multiple compression strategies. In most settings, our framework improves accuracy over post-shot and iterative baselines. Typical gains exceed 1%, while some configuration exhibits improvements above 15%. VCON is thus compatible with existing compression techniques and consistently improves performance across diverse tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑