arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2006.10598cs.LGcs.CLcs.CVstat.ML
收藏

神经参数分配搜索

Neural Parameter Allocation Search

Boston University(波士顿大学);ETH Zürich(苏黎世联邦理工学院);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)

Bryan A. Plummer, Nikoli Dryden, Julius Frost, Torsten Hoefler, Kate Saenko

摘要:

训练神经网络需要不断增加的内存量。参数共享可以减少内存和通信成本,但现有的方法假设网络有许多相同的层,并利用手工制作的共享策略,无法推广。我们介绍神经参数分配搜索(NPAS),一个新的任务,其目标是训练一个神经网络给定一个任意的,固定的参数预算。NPAS涵盖了两种低预算机制,产生紧凑的网络,以及一种新的高预算机制,其中可以添加额外的容量来提高性能,而不会增加推理FLOP。为了解决NPAS问题,我们引入了Shapeshifter Networks(SSN),它可以自动学习在网络中的何处以及如何共享参数,以支持任何参数预算,而无需对架构或损失函数进行任何更改。NPAS和SSN为解决广义参数共享提供了一个完整的框架,也可以与以前的工作相结合,以获得额外的性能增益。我们使用九种网络架构来展示我们方法的有效性,包括ImageNet分类和Transformers。

英文摘要:

Training neural networks requires increasing amounts of memory. Parameter sharing can reduce memory and communication costs, but existing methods assume networks have many identical layers and utilize hand-crafted sharing strategies that fail to generalize. We introduce Neural Parameter Allocation Search (NPAS), a novel task where the goal is to train a neural network given an arbitrary, fixed parameter budget. NPAS covers both low-budget regimes, which produce compact networks, as well as a novel high-budget regime, where additional capacity can be added to boost performance without increasing inference FLOPs. To address NPAS, we introduce Shapeshifter Networks (SSNs), which automatically learn where and how to share parameters in a network to support any parameter budget without requiring any changes to the architecture or loss function. NPAS and SSNs provide a complete framework for addressing generalized parameter sharing, and can also be combined with prior work for additional performance gains. We demonstrate the effectiveness of our approach using nine network architectures across four diverse tasks, including ImageNet classification and transformers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑