AuRA: 将音频理解内化到LLM中作为LoRA
AuRA: Internalizing Audio Understanding into LLMs as LoRA
AI总结：
提出AuRA方法，通过层间蒸馏将ASR编码器的语音表示内化到LoRA适配的LLM中，实现紧耦合的语音-语言联合建模和高效并行端到端推理，在多个基准上优于级联系统和现有适应方法。
摘要：
最近将大语言模型（LLM）扩展到语音输入的努力通常依赖于级联的ASR-LLM流水线、端到端语音-语言模型或基于桥接/蒸馏的适应方法。虽然这些路线分别重用了强大的预训练组件、实现了原生语音-语言交互或提供了轻量级适应，但它们常常遭受转录-接口延迟、昂贵的多模态训练或顺序语音-语言耦合的问题。为了解决这些限制，我们提出了AuRA，一种将音频编码能力蒸馏到LLM中的方法。具体来说，AuRA通过一个轻量级音频嵌入层将相同的语音输入馈送到ASR编码器（作为教师）和LoRA适配的LLM（作为学生），并使用逐层蒸馏将学生的隐藏状态与相应的教师表示对齐，从而将语音表示内化到轻量级的LLM侧适应中。与级联和串行桥接方法相比，AuRA实现了更紧密的语音-语言联合建模和高效的并行端到端推理，同时重用了预训练的语音和语言模型，而不需要大规模的多模态训练。在多个语音-语言基准上，AuRA在有效性和效率方面始终优于级联系统、语音到LLM适应基线以及大规模语音-语言和多模态模型。
英文摘要：
Recent efforts to extend large language models (LLMs) to speech inputs typically rely on cascaded ASR-LLM pipelines, end-to-end speech-language models, or bridge/distillation-based adaptation. While these routes respectively reuse strong pretrained components, enable native speech-language interaction, or offer lightweight adaptation, they often suffer from transcript-interface latency, costly multimodal training, or sequential speech-language coupling. To address these limitations, we present AuRA, a method that distills audio encoding capability into the LLM. Specifically, AuRA feeds the same speech input to an ASR encoder (as a teacher) and a LoRA-adapted LLM (as a student) through a lightweight audio embedding layer, and uses layer-wise distillation to align the student's hidden states with corresponding teacher representations, thereby internalizing speech representations into lightweight LLM-side adaptations. Compared with cascaded and serial bridge methods, AuRA enables tighter speech-language joint modeling and efficient parallel end-to-end inference, while also reusing pretrained speech and language models rather than requiring large-scale multimodal training. On multiple speech-language benchmarks, AuRA consistently outperforms cascaded systems, speech-to-LLM adaptation baselines, and large-scale speech-language and multimodal models in both effectiveness and efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
