arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.16892cs.LG
收藏

基于分块梯度匹配的稳健ASR训练数据子集选择

Partitioned Gradient Matching-based Data Subset Selection for Compute-Efficient Robust ASR Training

IBM Research(IBM研究院);Indian Institute of Technology Bombay(印度理工学院孟买分校);The University of Texas at Dallas(德克萨斯大学达拉斯分校)

Ashish Mittal, Durga Sivasubramanian, Rishabh Iyer, Preethi Jyothi, Ganesh Ramakrishnan

摘要:

培训最先进的ASR系统(如RNN-T)通常具有很高的相关财务和环境成本。如果选择的子集可以实现与使用整个数据集进行训练的性能相当,则使用训练数据的子集进行训练可以缓解此问题。虽然有许多数据子集选择(DSS)算法,但直接应用于RNN-T是困难的,特别是自适应和使用梯度等学习动态的DSS算法,因为RNN-T往往具有具有显着更大的内存占用的梯度。在本文中,我们提出了分区梯度匹配(PGM)一种新的分布式DSS算法,适用于大规模数据集,如用于训练RNN-T。通过在Librispeech 100 H和Librispeech 960 H上进行的大量实验,我们表明PGM在只有非常小的准确性下降(绝对WER差异低于1%)的情况下实现了3倍至6倍的加速。此外,即使在训练数据被噪声破坏的情况下,我们也证明了PGM的类似结果。

英文摘要:

Training state-of-the-art ASR systems such as RNN-T often has a high associated financial and environmental cost. Training with a subset of training data could mitigate this problem if the subset selected could achieve on-par performance with training with the entire dataset. Although there are many data subset selection(DSS) algorithms, direct application to the RNN-T is difficult, especially the DSS algorithms that are adaptive and use learning dynamics such as gradients, as RNN-T tend to have gradients with a significantly larger memory footprint. In this paper, we propose Partitioned Gradient Matching (PGM) a novel distributable DSS algorithm, suitable for massive datasets like those used to train RNN-T. Through extensive experiments on Librispeech 100H and Librispeech 960H, we show that PGM achieves between 3x to 6x speedup with only a very small accuracy degradation (under 1% absolute WER difference). In addition, we demonstrate similar results for PGM even in settings where the training data is corrupted with noise.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑