窄带语音通信的流式神经压缩

中文摘要 AI 辅助

在资源受限的边缘设备上进行低比特率语音通信，由于严格的计算、内存和带宽限制，仍然具有挑战性。我们提出了TinyCall，一种轻量级神经音频编解码器，专为在低功耗平台（如ESP32微控制器和树莓派）上进行实时语音通信而设计。所提出的系统针对应急通信和其他带宽受限场景，同时保持语音可懂度、说话人身份和声音表现力。为了实现高效部署，我们提出了一种极简神经音频编解码器架构，以及一个通过伪前瞻解码和解码器输入缓存将因果训练编解码器转换为真正可流式编解码器的框架。我们进一步用残差有限标量量化（RFSQ）替代传统的残差向量量化（RVQ），以降低边缘处理器上的推理复杂度，并采用渐进式三阶段训练策略，在潜在量化下实现稳定优化。基于MFCC的感知损失鼓励保留说话人特征，包括谐波结构和声音音色。实验结果表明，在树莓派3上实现了实时操作，同时以低至2.3 kbps的比特率实现了可懂的语音重建。所提出的方法表明，在高度资源受限的边缘设备上实现实用的神经语音通信是可行的。

英文摘要

Low-bitrate speech communication on resource-constrained edge devices remains challenging due to stringent computational, memory, and bandwidth constraints. We present TinyCall, a lightweight neural audio codec designed for real-time speech communication on low-power platforms such as the ESP32 microcontroller and Raspberry Pi. The proposed system targets emergency communication and other bandwidth-limited scenarios while preserving speech intelligibility, speaker identity, and vocal expressiveness. To enable efficient deployment, we propose a minimal neural audio codec architecture together with a framework for converting a causally trained codec into a truly streamable codec through pseudo-lookahead decoding and decoder-input caching. We further replace conventional residual vector quantization (RVQ) with Residual Finite Scalar Quantization (RFSQ) to reduce inference complexity on edge processors and employ a progressive three-stage training strategy for stable optimization under latent quantization. An MFCC-based perceptual loss encourages preservation of speaker characteristics, including harmonic structure and vocal timbre. Experimental results demonstrate real-time operation on a Raspberry Pi 3 while achieving intelligible speech reconstruction at bitrates as low as 2.3 kbps. The proposed approach demonstrates that practical neural speech communication is feasible on highly resource-constrained edge devices.

发表机构

• University of Maryland, College Park（马里兰大学帕克分校）

机构由 AI 辅助整理，请以论文原文为准。