当行动教你思考：通过强化学习在对话代理中实现推理-行动协同
When Actions Teach You to Think: Reasoning-Action Synergy via Reinforcement Learning in Conversational Agents
AI总结：
通过强化学习实现对话代理中推理与行动的协同，提升模型推理质量和工具调用精度。
摘要：
监督微调（SFT）已成为提高大型语言模型（LLMs）在下游任务性能的最有效方法之一。然而，当底层数据分布发生变化时，SFT可能会遇到泛化困难，即使新数据不完全超出训练领域。最近的推理导向模型如o1和R1在非推理模型上表现出一致的提升，突显了推理在提高泛化能力和可靠性方面的重要性。然而，收集高质量的推理轨迹进行SFT仍然具有挑战性——注释成本高、主观且难以扩展。为了解决这一限制，我们利用强化学习（RL）使模型能够直接从任务结果中学习推理策略。我们提出了一种流程，其中LLMs生成推理步骤，以指导工具的调用（例如，函数调用）和对话代理的最终答案生成。我们的方法采用组相对策略优化（GRPO），奖励设计围绕工具准确性和答案正确性，使模型能够迭代地优化其推理和行动。实验结果表明，我们的方法提高了推理质量和工具调用的精确度，相较于无显式思考训练的SFT模型提升了1.5%，相较于基础的Qwen3-1.7B模型提升了40%。这些发现展示了通过强化学习统一推理和行动学习的潜力，以构建更强大且更具泛化能力的对话代理。
英文摘要：
Supervised fine-tuning (SFT) has emerged as one of the most effective ways to improve the performance of large language models (LLMs) in downstream tasks. However, SFT can have difficulty generalizing when the underlying data distribution changes, even when the new data does not fall completely outside the training domain. Recent reasoning-focused models such as o1 and R1 have demonstrated consistent gains over their non-reasoning counterparts, highlighting the importance of reasoning for improved generalization and reliability. However, collecting high-quality reasoning traces for SFT remains challenging -- annotations are costly, subjective, and difficult to scale. To address this limitation, we leverage Reinforcement Learning (RL) to enable models to learn reasoning strategies directly from task outcomes. We propose a pipeline in which LLMs generate reasoning steps that guide both the invocation of tools (e.g., function calls) and the final answer generation for conversational agents. Our method employs Group Relative Policy Optimization (GRPO) with rewards designed around tool accuracy and answer correctness, allowing the model to iteratively refine its reasoning and actions. Experimental results demonstrate that our approach improves both the quality of reasoning and the precision of tool invocations, achieving a 1.5% relative improvement over the SFT model (trained without explicit thinking) and a 40% gain compared to the base of the vanilla Qwen3-1.7B model. These findings demonstrate the promise of unifying reasoning and action learning through RL to build more capable and generalizable conversational agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
