面向可扩展ASR模型压缩的正交Softmax高效超网训练
Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compression
AI总结:
本文提出OrthoSoftmax方法,在超网训练中高效识别最优子网,实现可扩展ASR模型压缩,实验表明FLOPs感知组件级选择性能最佳。
摘要:
ASR系统部署在多样化的环境中,每种环境都有特定的硬件约束。我们使用超网训练来联合训练多个不同大小的编码器,从而能够动态调整模型大小以适应硬件约束,而无需进行冗余训练。此外,我们引入了一种名为OrthoSoftmax的新方法,该方法应用多个正交softmax函数,在超网内高效地识别最优子网,避免了资源密集型的搜索。该方法还通过允许基于各种标准和粒度级别进行选择,实现了更灵活、更精确的子网选择。我们在Librispeech和TED-LIUM-v2上使用CTC的实验结果表明,基于FLOPs感知的组件级选择取得了最佳整体性能。在单个任务中执行相同数量的训练更新时,所有模型大小的WER与单独训练的模型相当或略好。此外,我们分析了所选组件中的模式,并揭示了有趣的见解。
英文摘要:
ASR systems are deployed across diverse environments, each with specific hardware constraints. We use supernet training to jointly train multiple encoders of varying sizes, enabling dynamic model size adjustment to fit hardware constraints without redundant training. Moreover, we introduce a novel method called OrthoSoftmax, which applies multiple orthogonal softmax functions to efficiently identify optimal subnets within the supernet, avoiding resource-intensive search. This approach also enables more flexible and precise subnet selection by allowing selection based on various criteria and levels of granularity. Our results with CTC on Librispeech and TED-LIUM-v2 show that FLOPs-aware component-wise selection achieves the best overall performance. With the same number of training updates from one single job, WERs for all model sizes are comparable to or slightly better than those of individually trained models. Furthermore, we analyze patterns in the selected components and reveal interesting insights.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。