arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.17065eess.AScs.CLcs.LG
收藏

用 Pilot Inference 为语音理解加速

Turbocharge Speech Understanding with Pilot Inference

University of Virginia(弗吉尼亚大学)

Rongxiang Wang, Felix Xiaozhu Lin

AI总结:

本文提出PASU原型,通过延迟上下文化、pilot inference和自回归出口等技术,加速资源受限边缘设备上的语音理解,在保持SOTA精度的同时将端到端延迟和卸载需求各降低2倍。

摘要:

现代语音理解(SU)运行复杂的流水线:摄取流式语音输入时,该流水线反复执行基于编码器-解码器的深度神经网络;通过这种方式,流水线生成暂定输出(称为假设),并定期对这些假设评分。本文旨在加速资源受限边缘设备上的SU。它采用混合方法:加速设备端执行;卸载超出设备能力的输入。虽然该方法广为人知,但我们用新技术应对SU的独特挑战:(1)延迟上下文化,并行执行模型的注意力编码器与输入摄取;(2)pilot inference,缓解SU流水线的时间负载不均衡;(3)自回归出口,基于pilot inference和假设评估卸载决策。我们的技术与现有语音模型、流水线和框架兼容;可独立或组合应用。原型PASU在6-8核Arm平台测试:实现SOTA精度;端到端延迟降低2倍,卸载需求减少2倍。

英文摘要:

Modern speech understanding (SU) runs a sophisticated pipeline: ingesting streaming voice input, the pipeline executes encoder-decoder based deep neural networks repeatedly; by doing so, the pipeline generates tentative outputs (called hypotheses), and periodically scores the hypotheses. This paper sets to accelerate SU on resource-constrained edge devices. It takes a hybrid approach: to speed up on-device execution; to offload inputs that are beyond the device's capacity. While the approach is well-known, we address SU's unique challenges with novel techniques: (1) late contextualization, which executes a model's attentive encoder in parallel to the input ingestion; (2) pilot inference, which mitigates the SU pipeline's temporal load imbalance; (3) autoregression offramps, which evaluate offloading decisions based on pilot inferences and hypotheses. Our techniques are compatible with existing speech models, pipelines, and frameworks; they can be applied independently or in combination. Our prototype, called PASU, is tested on Arm platforms with 6 - 8 cores: it delivers SOTA accuracy; it reduces the end-to-end latency by 2x and reduces the offloading needs by 2x.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑