arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.19377cs.SDcs.CLcs.LGeess.AS
收藏

SELMA:一种面向虚拟助手交互的语音赋能语言模型

SELMA: A Speech-Enabled Language Model for Virtual Assistant Interactions

TH Nürnberg(纽伦堡应用技术大学);Apple(苹果公司)

Dominik Wagner, Alexander Churchill, Siddharth Sigtia, Erik Marchi

AI总结:

本文提出 SELMA,一种将音频和文本集成到 LLM 的端到端语音赋能模型,通过低秩适配和特征池化同时处理虚拟助手多个任务,在 VT 检测、DDSD 和 ASR 上显著简化流程并提升性能。

摘要:

在这项工作中,我们提出并评估了 SELMA,一种面向虚拟助手交互的语音赋能语言模型,它将音频和文本作为输入集成到大型语言模型(LLM)中。SELMA 被设计为在单个端到端模型中同时处理与虚拟助手交互相关的三个主要任务和两个辅助任务。我们采用低秩适配模块对音频编码器和 LLM 进行参数高效训练。此外,我们实现了一种特征池化策略,使系统能够识别全局模式,并在较少依赖单个序列元素的任务上提高准确性。在语音触发(VT)检测、设备定向语音检测(DDSD)和自动语音识别(ASR)上的实验结果表明,我们的方法既显著简化了虚拟助手的典型输入处理流程,又相比针对每个单独任务的专用模型提高了性能。SELMA 在 VT 检测任务上实现了 64% 的相对等错误率改进,在 DDSD 上实现了 22% 的改进,同时取得了接近基线的词错误率。

英文摘要:

In this work, we present and evaluate SELMA, a Speech-Enabled Language Model for virtual Assistant interactions that integrates audio and text as inputs to a Large Language Model (LLM). SELMA is designed to handle three primary and two auxiliary tasks related to interactions with virtual assistants simultaneously within a single end-to-end model. We employ low-rank adaptation modules for parameter-efficient training of both the audio encoder and the LLM. Additionally, we implement a feature pooling strategy enabling the system to recognize global patterns and improve accuracy on tasks less reliant on individual sequence elements. Experimental results on Voice Trigger (VT) detection, Device-Directed Speech Detection (DDSD), and Automatic Speech Recognition (ASR), demonstrate that our approach both simplifies the typical input processing pipeline of virtual assistants significantly and also improves performance compared to dedicated models for each individual task. SELMA yields relative Equal-Error Rate improvements of 64% on the VT detection task, and 22% on DDSD, while also achieving word error rates close to the baseline.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑