arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.00356cs.LGcs.CV
收藏

增强网络参数化中神经表示的准确性与参数效率

Enhancing Accuracy and Parameter-Efficiency of Neural Representations for Network Parameterization

Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室)

Hongjun Choi, Jayaraman J. Thiagarajan, Ruben Glatt, Shusen Liu

AI总结:

本文研究神经网络权重参数化中准确性与参数效率的权衡,发现仅靠权重重建即可恢复模型准确性,并提出解耦辅助目标的新训练方案,以同时提升模型准确性与预测器参数效率。

摘要:

在这项工作中,我们研究了使用预测器网络对神经网络权重进行参数化时,准确性与参数效率之间的基本权衡。我们提出了一个令人惊讶的发现:当恢复原始模型准确性是唯一目标时,仅通过权重重建目标即可有效实现。此外,我们探讨了在参数效率约束下改进权重重建的潜在因素,并提出了一种新颖的训练方案,该方案将重建目标与知识蒸馏等辅助目标解耦,从而与最先进的方法相比带来了显著改进。最后,这些结果为更实际的场景铺平了道路,在这些场景中,需要同时实现模型准确性和预测器网络参数效率的提升。

英文摘要:

In this work, we investigate the fundamental trade-off regarding accuracy and parameter efficiency in the parameterization of neural network weights using predictor networks. We present a surprising finding that, when recovering the original model accuracy is the sole objective, it can be achieved effectively through the weight reconstruction objective alone. Additionally, we explore the underlying factors for improving weight reconstruction under parameter-efficiency constraints, and propose a novel training scheme that decouples the reconstruction objective from auxiliary objectives such as knowledge distillation that leads to significant improvements compared to state-of-the-art approaches. Finally, these results pave way for more practical scenarios, where one needs to achieve improvements on both model accuracy and predictor network parameter-efficiency simultaneously.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑