假世界，真实技能：利用合成任务、模拟环境和评分标准奖励构建小型代理语言模型
Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards
AI总结：
SYNTHAGENT框架通过合成多样化任务和模拟环境，结合评分标准奖励，提升小型LLM在数学、搜索和工具使用等任务中的代理能力。
摘要：
小型LLM往往难以匹配大而昂贵模型的代理能力。尽管强化学习可以帮助，但进展受到两个结构性瓶颈的限制：现有的开源代理训练数据任务种类狭窄且容易解决；现实世界API缺乏多样性且在大规模强化学习展开过程中不稳定。我们通过SYNTHAGENT框架解决这些挑战，该框架联合合成多样化工具使用训练数据和模拟完整环境。具体而言，一个强大的教师模型会创建新颖的任务和工具生态系统，然后将其重写为故意不明确的指令。这迫使代理主动查询用户以获取缺失的细节。在处理合成任务时，基于LLM的用户模拟器提供用户私有信息，而假工具系统提供稳定的工具响应。在奖励方面，基于所需子目标、用户-代理互动和禁止行为构建任务级评分标准。在数学、搜索和工具使用等14个具有挑战性的数据集中，使用我们合成数据训练的模型取得了显著提升，小型模型在较大基线模型之上表现更优。
英文摘要：
Small LLMs often struggle to match the agentic capabilities of large, costly models. While reinforcement learning can help, progress has been limited by two structural bottlenecks: existing open-source agentic training data are narrow in task variety and easily solved; real-world APIs lack diversity and are unstable for large-scale reinforcement learning rollout processes. We address these challenges with SYNTHAGENT, a framework that jointly synthesizes diverse tool-use training data and simulates complete environments. Specifically, a strong teacher model creates novel tasks and tool ecosystems, then rewrites them into intentionally underspecified instructions. This compels agents to actively query users for missing details. When handling synthetic tasks, an LLM-based user simulator provides user-private information, while a mock tool system delivers stable tool responses. For rewards, task-level rubrics are constructed based on required subgoals, user-agent interactions, and forbidden behaviors. Across 14 challenging datasets in math, search, and tool use, models trained on our synthetic data achieve substantial gains, with small models outperforming larger baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
