利用GRPO推进语音感知大语言模型的语音理解能力
Advancing Speech Understanding in Speech-Aware Language Models with GRPO
AI总结：
本文提出用GRPO结合BLEU奖励信号训练语音感知大语言模型，在口语问答和语音翻译等开放式任务上超越标准SFT，并探索离策略样本的改进潜力。
摘要：
本文提出一种基于分组相对策略优化（GRPO）的方法，用于在开放式语音理解任务（如口语问答和自动语音翻译）上训练语音感知大语言模型（SALLMs）。SALLMs已被证明在语音理解任务中非常有效。GRPO近期因其训练大语言模型的高效性而受到关注，此前已有工作探索其在SALLMs上的应用，但主要集中于多项选择题任务。在此基础上，我们聚焦于更能体现模型生成能力的开放式任务。我们的方法利用GRPO并以BLEU作为奖励信号来优化SALLMs，并通过实验证明该方法在多个关键指标上超越了标准监督微调（SFT）。最后，我们探索了在这些任务中将离策略样本纳入GRPO的潜力，为进一步改进和后续研究指明了方向。
英文摘要：
In this paper, we introduce a Group Relative Policy Optimization (GRPO)-based method for training Speech-Aware Large Language Models (SALLMs) on open-format speech understanding tasks, such as Spoken Question Answering and Automatic Speech Translation. SALLMs have proven highly effective for speech understanding tasks. GRPO has recently gained traction for its efficiency in training LLMs, and prior work has explored its application to SALLMs, primarily in multiple-choice tasks. Building on this, we focus on open-format tasks that better reflect the generative abilities of the models. Our approach leverages GRPO with BLEU as the reward signal to optimize SALLMs, and we demonstrate empirically that it surpasses standard SFT across several key metrics. Finally, we explore the potential of incorporating off-policy samples within GRPO for these tasks, highlighting avenues for further improvement and further research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
