从零构建企业级实时语音代理：技术教程
Building Enterprise Realtime Voice Agents from Scratch: A Technical Tutorial
AI总结：
本文介绍如何从零构建企业级实时语音代理，通过Deepgram、vLLM和ElevenLabs实现端到端流程，展示最佳实践与代码实现。
摘要：
我们提供了一项技术教程，介绍如何从基础原理构建企业级实时语音代理。尽管端到端的语音到语音模型可能最终提供最佳延迟，但完全自托管的端到端解决方案尚未可用。我们评估了最接近的候选者Qwen3-Omni，其云服务DashScope实时API实现约702ms的音频到音频延迟，但无法自托管；其本地vLLM部署仅支持Thinker（从音频生成文本，516ms），不支持Talker（音频合成）；其本地Transformers部署运行完整流程但延迟约146秒，远不足以实现实时。因此，STT→LLM→TTS的级联流式管道仍然是自托管实时语音代理的实用架构。本文通过Deepgram（流式STT）、vLLM服务的LLMs（流式文本生成）和ElevenLabs（流式TTS）构建完整语音代理，实现了755ms（最佳情况729ms）的首次音频时间，支持完整功能调用。本文发布完整的代码库作为9章渐进教程，为每个组件提供经过测试的代码。
英文摘要：
We present a technical tutorial for building enterprise-grade realtime voice agents from first principles. While end-to-end speech-to-speech models may ultimately provide the best latency for voice agents, fully self-hosted end-to-end solutions are not yet available. We evaluate the closest candidate, Qwen3-Omni, across three configurations: its cloud-only DashScope Realtime API achieves $\sim$702ms audio-to-audio latency with streaming, but is not self-hostable; its local vLLM deployment supports only the Thinker (text generation from audio, 516ms), not the Talker (audio synthesis); and its local Transformers deployment runs the full pipeline but at $\sim$146s -- far too slow for realtime. The cascaded streaming pipeline (STT $\rightarrow$ LLM $\rightarrow$ TTS) therefore remains the practical architecture for self-hosted realtime voice agents, and the focus of this tutorial. We build a complete voice agent using Deepgram (streaming STT), vLLM-served LLMs with function calling (streaming text generation), and ElevenLabs (streaming TTS), achieving a measured time-to-first-audio of 755ms (best case 729ms) with full function calling support. We release the full codebase as a 9-chapter progressive tutorial with working, tested code for every component.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
