用于设备上定向听觉的混合神经网络
Hybrid Neural Networks for On-device Directional Hearing
摘要:
设备上的定向听觉需要从给定方向进行音频声源分离,同时实现严格的人类感知不到的延迟要求。虽然神经网络可以实现比传统波束形成器更好的性能,但所有现有模型都无法支持计算受限的可穿戴设备上的低延迟因果推理。我们提出了DeepBeam,这是一种将传统波束形成器与自定义轻量级神经网络相结合的混合模型。前者减少了后者的计算负担,也提高了其通用性,而后者旨在进一步减少内存和计算开销,以实现实时和低延迟操作。我们的评估显示,在合成数据上的性能与最先进的因果推理模型相当,同时实现了模型大小减少5倍,每秒计算减少4倍,处理时间减少5倍,并更好地推广到实际硬件数据。此外,我们的实时混合模型在为低功耗可穿戴设备设计的移动CPU上运行时间为8 ms,并实现了17.5 ms的端到端延迟。
英文摘要:
On-device directional hearing requires audio source separation from a given direction while achieving stringent human-imperceptible latency requirements. While neural nets can achieve significantly better performance than traditional beamformers, all existing models fall short of supporting low-latency causal inference on computationally-constrained wearables. We present DeepBeam, a hybrid model that combines traditional beamformers with a custom lightweight neural net. The former reduces the computational burden of the latter and also improves its generalizability, while the latter is designed to further reduce the memory and computational overhead to enable real-time and low-latency operations. Our evaluation shows comparable performance to state-of-the-art causal inference models on synthetic data while achieving a 5x reduction of model size, 4x reduction of computation per second, 5x reduction in processing time and generalizing better to real hardware data. Further, our real-time hybrid model runs in 8 ms on mobile CPUs designed for low-power wearable devices and achieves an end-to-end latency of 17.5 ms.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。