TinySpeech：边缘设备上深度语音识别神经网络的注意力凝聚器
TinySpeech: Attention Condensers for Deep Speech Recognition Neural Networks on Edge Devices
摘要：
深度学习的进步已经在众多语音识别任务中实现了最先进的性能。然而，用于设备上语音识别的深度神经网络的广泛部署仍然是一个挑战，特别是在存储器和计算资源高度受限的边缘场景中（例如，低功耗嵌入式设备）或专用于语音识别的存储器和计算预算较低的情况（例如，执行除语音识别之外的许多任务的移动设备）。在这项研究中，我们引入了注意力凝聚器的概念，用于构建低占用空间，高效的深度神经网络，用于边缘上的设备上语音识别。注意冷凝器是一种自我注意机制，它学习并产生表征联合局部和跨通道激活关系的浓缩嵌入，并相应地执行选择性注意。为了说明其有效性，我们介绍了TinySpeech，低精度深度神经网络，主要包括使用机器驱动的设计探索策略为设备上的语音识别量身定制的注意力冷凝器，其中一个专门针对微控制器操作约束。在有限词汇量语音识别的Google Speech Commands基准数据集上的实验结果表明，与以前的工作相比，TinySpeech网络实现了显着降低的架构复杂性（$507\times$更少的参数），更低的计算复杂性（$48\times$更少的乘加运算）和更低的存储要求（$2028\times$更低的权重内存要求）。这些结果不仅证明了注意力冷凝器在构建高效的设备上语音识别网络方面的有效性，还阐明了其在边缘加速深度学习和增强TinyML应用程序的潜力。
英文摘要：
Advances in deep learning have led to state-of-the-art performance across a multitude of speech recognition tasks. Nevertheless, the widespread deployment of deep neural networks for on-device speech recognition remains a challenge, particularly in edge scenarios where the memory and computing resources are highly constrained (e.g., low-power embedded devices) or where the memory and computing budget dedicated to speech recognition is low (e.g., mobile devices performing numerous tasks besides speech recognition). In this study, we introduce the concept of attention condensers for building low-footprint, highly-efficient deep neural networks for on-device speech recognition on the edge. An attention condenser is a self-attention mechanism that learns and produces a condensed embedding characterizing joint local and cross-channel activation relationships, and performs selective attention accordingly. To illustrate its efficacy, we introduce TinySpeech, low-precision deep neural networks comprising largely of attention condensers tailored for on-device speech recognition using a machine-driven design exploration strategy, with one tailored specifically with microcontroller operation constraints. Experimental results on the Google Speech Commands benchmark dataset for limited-vocabulary speech recognition showed that TinySpeech networks achieved significantly lower architectural complexity (as much as $507\times$ fewer parameters), lower computational complexity (as much as $48\times$ fewer multiply-add operations), and lower storage requirements (as much as $2028\times$ lower weight memory requirements) when compared to previous work. These results not only demonstrate the efficacy of attention condensers for building highly efficient networks for on-device speech recognition, but also illuminate its potential for accelerating deep learning on the edge and empowering TinyML applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
