探索基于大语言模型的ASR与TTS系统的强化学习
Explore the Reinforcement Learning for the LLM based ASR and TTS system
AI总结：
本文提出一个面向音频大语言模型的轻量级强化学习框架，在ASR中探索GRPO奖励函数与数据构建，在TTS中结合GRPO与DiffRO，证明有限数据和步数下RL仍能显著提升ASR与TTS性能。
摘要：
近年来，大语言模型（LLMs）在自动语音识别（ASR）和文本到语音（TTS）系统中发挥了重要作用。尽管强化学习（RL）已显著提升了LLM在基于文本的任务上的性能，但由于训练基于音频的模型的复杂性，其在ASR和TTS中的应用仍未得到充分探索。在本研究中，我们提出了一个轻量级RL框架，专门针对能够处理音频输入并生成音频输出的基于音频的LLM。基于该框架，我们评估了强化学习在ASR和TTS任务上的有效性。对于ASR任务，我们在分组相对策略优化（GRPO）框架内尝试了不同的基于规则的奖励函数，并研究了RL数据构建的影响。对于TTS任务，我们将GRPO与可微奖励优化（DiffRO）进行了比较，并进一步结合这两种方法以实现性能提升。我们的实验表明，即使在有限的训练数据和较少的优化步数下，RL也能显著提升ASR和TTS系统的性能。
英文摘要：
In recent years, large language models (LLMs) have played an important role in automatic speech recognition (ASR) and text-to-speech (TTS) systems. While reinforcement learning (RL) has significantly enhanced LLM performance in text-based tasks, its application to ASR and TTS remains underexplored due to the complexity of training audio-based models. In this study, we propose a lightweight RL framework tailored for audio-based LLMs that can process audio inputs and generate audio outputs. Based on this framework, we evaluate the effectiveness of reinforcement learning on both ASR and TTS tasks. For the ASR task, we experiment with different rule-based reward functions within the Group Relative Policy Optimization (GRPO) framework and investigate the impact of RL data construction. For the TTS task, we compare GRPO with Differentiable Reward Optimization (DiffRO) and further combine the two approaches to achieve improved performance. Our experiments demonstrate that RL can significantly enhance the performance of both ASR and TTS systems, even with limited training data and a small number of optimization steps.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
