稀疏奖励可以自训练对话智能体
Sparse Rewards Can Self-Train Dialogue Agents
AI总结：
本文提出JOSH自对齐算法，利用稀疏奖励模拟环境ToolWOZ让LLM智能体在无人类反馈条件下自主提升工具调用能力，同时保持通用能力。
摘要：
近年来，最先进（SOTA）大语言模型（LLM）智能体的进展，尤其是在多轮对话任务中的进展，主要受监督微调和高质量人类反馈驱动。然而，随着基础LLM模型持续改进，获取有意义的人类反馈变得日益困难且成本高昂。在某些领域，基础LLM智能体最终可能超越人类能力，使得传统的反馈驱动方法不再可行。本文提出一种新颖的自我改进范式，使LLM智能体能够在没有外部人类反馈的情况下自主提升性能。我们的方法JOSH（Juxtaposed Outcomes for Simulation Harvesting，并置结果模拟采集）是一种自对齐算法，利用稀疏奖励模拟环境提取理想行为，并在LLM自身输出上进一步训练。我们提出了ToolWOZ，一个源自MultiWOZ的稀疏奖励工具调用模拟环境。我们证明，使用JOSH训练的模型（包括小型和前沿模型）显著改善了基于工具的交互，同时在多个基准上保持了通用模型能力。我们的代码和数据已在GitHub上公开：https://github.com/asappresearch/josh-llm-simulation-training。
英文摘要：
Recent advancements in state-of-the-art (SOTA) Large Language Model (LLM) agents, especially in multi-turn dialogue tasks, have been primarily driven by supervised fine-tuning and high-quality human feedback. However, as base LLM models continue to improve, acquiring meaningful human feedback has become increasingly challenging and costly. In certain domains, base LLM agents may eventually exceed human capabilities, making traditional feedback-driven methods impractical. In this paper, we introduce a novel self-improvement paradigm that empowers LLM agents to autonomously enhance their performance without external human feedback. Our method, Juxtaposed Outcomes for Simulation Harvesting (JOSH), is a self-alignment algorithm that leverages a sparse reward simulation environment to extract ideal behaviors and further train the LLM on its own outputs. We present ToolWOZ, a sparse reward tool-calling simulation environment derived from MultiWOZ. We demonstrate that models trained with JOSH, both small and frontier, significantly improve tool-based interactions while preserving general model capabilities across diverse benchmarks. Our code and data are publicly available on GitHub at https://github.com/asappresearch/josh-llm-simulation-training
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
