RePlay：基于检索的语音回放用于多轮口语对话

中文摘要 AI 辅助

许多语音交互应用需要对响应的内容和表达方式进行精确控制，通常使用预先录制的语音片段。最近的全双工模型虽然能以低延迟响应，但无法保证确切的内容或重现特定的录制效果，而级联系统虽然可以限制为预定义响应，却以额外的延迟为代价。我们提出了RePlay，一种从PersonaPlex改编的口语对话系统，通过检索并播放预先录制的语音片段来处理多轮对话。利用探测方法，我们确定了即将到来的响应变得可恢复的层和帧，并将该隐藏状态作为检索查询。RePlay仅保留到该点的层，并用轻量级的轮流发言和检索头取代文本和语音生成。在模拟的多轮面试中，RePlay的中位延迟达到383毫秒，比具有相当对话质量的ASR-LLM级联系统低3至7倍，但代价是精确线路准确率较低。在一项用户研究中，参与者偏好RePlay的比例为63%，而偏好带有小型LLM的快速级联系统的比例为12%（p = 0.008），并且与带有更强LLM的较慢级联系统相比，显示出非显著的偏好（46%对21%）。

英文摘要

Many voice interaction applications require exact control over both the content and delivery of responses, typically using pre-recorded lines. Recent full-duplex models respond with low latency but cannot guarantee exact content or reproduce a specific recorded performance, while cascaded systems can be constrained to predefined responses at the cost of additional latency. We propose RePlay, a spoken dialogue system adapted from PersonaPlex that handles multi-turn conversations by retrieving and playing pre-recorded lines. Using probing, we identify the layer and frame at which the upcoming response becomes recoverable, and use this hidden state as the retrieval query. RePlay retains only the layers up to that point and replaces text and speech generation with lightweight turn-taking and retrieval heads. In simulated multi-turn interviews, RePlay reaches a median latency of 383 ms, 3 to 7 times lower than ASR-LLM cascades of comparable dialogue quality, at the cost of lower exact-line accuracy. In a user study, participants preferred RePlay in 63% of ratings versus 12% for a fast cascade with a small LLM (p = 0.008), and showed a non-significant preference (46% vs. 21%) over a slower cascade with a stronger LLM.

发表机构

• Disney Research Imagineering（迪士尼幻想工程研究院）

机构由 AI 辅助整理，请以论文原文为准。