Audio-Thinker：通过强化学习引导音频语言模型何时及如何思考
Audio-Thinker: Guiding Audio Language Model When and How to Think via Reinforcement Learning
AI总结：
本文提出强化学习框架Audio-Thinker，通过引入自适应思考准确度奖励和外部奖励模型评估推理一致性，引导大型音频语言模型动态调整推理策略，显著提升了其在音频问答等任务上的推理与泛化能力。
摘要：
大型语言模型、多模态大型语言模型和大型音频语言模型（LALMs）的最新进展，通过带有基于规则奖励的强化学习显著提升了其推理能力。然而，显式推理过程尚未在音频问答中展现出显著益处，有效利用深度推理仍是一个开放性挑战，LALMs在听觉-语言推理方面仍达不到人类水平。为解决这些局限性，我们提出了Audio-Thinker，这是一个旨在增强LALMs推理能力的强化学习框架，重点提升适应性、一致性和有效性。我们的方法引入了自适应思考准确度奖励，使模型能够根据任务复杂度动态调整其推理策略。此外，我们结合了外部奖励模型来评估推理过程的整体一致性和质量，并辅以基于思考的奖励，帮助模型在训练期间区分有效和有缺陷的推理路径。实验结果表明，我们的Audio-Thinker模型在各种基准任务上优于现有的面向推理的LALMs，展现出卓越的推理和泛化能力。
英文摘要：
Recent advancements in large language models, multimodal large language models, and large audio language models (LALMs) have significantly improved their reasoning capabilities through reinforcement learning with rule-based rewards. However, the explicit reasoning process has yet to show significant benefits for audio question answering, and effectively leveraging deep reasoning remains an open challenge, with LALMs still falling short of human-level auditory-language reasoning. To address these limitations, we propose Audio-Thinker, a reinforcement learning framework designed to enhance the reasoning capabilities of LALMs, with a focus on improving adaptability, consistency, and effectiveness. Our approach introduces an adaptive think accuracy reward, enabling the model to adjust its reasoning strategies based on task complexity dynamically. Furthermore, we incorporate an external reward model to evaluate the overall consistency and quality of the reasoning process, complemented by think-based rewards that help the model distinguish between valid and flawed reasoning paths during training. Experimental results demonstrate that our Audio-Thinker model outperforms existing reasoning-oriented LALMs across various benchmark tasks, exhibiting superior reasoning and generalization capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
