arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.13174cs.LGcs.IR
收藏

压缩模型并非大型模型的微型版本

Compressed models are NOT miniature versions of large models

Indian Institute of Technology Guwahati(印度古瓦哈蒂理工学院)

Rohit Raj Rai, Rishant Pal, Amit Awekar

AI总结:

本文质疑压缩模型是大型模型微型版本的普遍假设,通过比较预测误差、数据表示、数据分布和对抗攻击脆弱性四个特征,发现压缩模型与大型模型及彼此间均存在显著差异,并指出其替代带来的副作用。

摘要:

大型神经模型在部署前通常会被压缩。模型压缩因许多实际原因而必要,例如推理延迟、内存占用和能耗。压缩模型被假定为对应大型神经模型的微型版本。然而,我们在工作中质疑了这一信念。我们使用四个模型特征来比较压缩模型与对应的大型神经模型:预测误差、数据表示、数据分布以及对对抗攻击的脆弱性。我们使用BERT-large模型及其五个压缩版本进行了实验。对于所有四个模型特征,压缩模型与BERT-large模型显著不同。即使在压缩模型之间,它们在所有四个模型特征上也彼此不同。除了预期的模型性能损失外,使用压缩模型替代大型神经模型还存在重大的副作用。

英文摘要:

Large neural models are often compressed before deployment. Model compression is necessary for many practical reasons, such as inference latency, memory footprint, and energy consumption. Compressed models are assumed to be miniature versions of corresponding large neural models. However, we question this belief in our work. We compare compressed models with corresponding large neural models using four model characteristics: prediction errors, data representation, data distribution, and vulnerability to adversarial attack. We perform experiments using the BERT-large model and its five compressed versions. For all four model characteristics, compressed models significantly differ from the BERT-large model. Even among compressed models, they differ from each other on all four model characteristics. Apart from the expected loss in model performance, there are major side effects of using compressed models to replace large neural models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑