arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.03076cs.LGcs.AR
收藏

用于深度学习硬件精度恢复的敏感度感知微调

Sensitivity-Aware Finetuning for Accuracy Recovery on Deep Learning Hardware

Lightmatter(Lightmatter公司)

Lakshmi Nair, Darius Bunandar

AI总结:

针对模数硬件上噪声注入训练成本高的问题,本文提出SAFT方法,通过识别并冻结噪声敏感层来降低微调开销,同时保持相当精度并实现2至8倍加速。

摘要:

在存在量化和模拟噪声的模数硬件上,恢复模型精度的现有方法包括噪声注入训练。然而,即使从预训练模型开始,该方法在实践中也可能较慢,并产生较高计算成本。我们提出敏感度感知微调(Sensitivity-Aware Finetuning,SAFT)方法,该方法识别模型中的噪声敏感层,并利用该信息冻结特定层以进行噪声注入训练。结果表明,SAFT达到了与噪声注入训练相当的精度,并且速度快2倍至8倍。

英文摘要:

Existing methods to recover model accuracy on analog-digital hardware in the presence of quantization and analog noise include noise-injection training. However, it can be slow in practice, incurring high computational costs, even when starting from pretrained models. We introduce the Sensitivity-Aware Finetuning (SAFT) approach that identifies noise sensitive layers in a model, and uses the information to freeze specific layers for noise-injection training. Our results show that SAFT achieves comparable accuracy to noise-injection training and is 2x to 8x faster.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑