SCOPE：面向开放式任务的协同演化策略自我对弈
SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Tasks
AI总结：
提出SCOPE框架，通过协同演化挑战者和求解者策略，实现无数据自我对弈，在开放式任务上提升性能并超越依赖人工提示的方法。
摘要：
自我对弈可以在没有外部监督的情况下训练语言模型。然而，现有方法需要可规则检查的答案，使得开放式任务依赖于精心设计的提示或前沿模型作为评判者。我们引入了SCOPE，一个无数据的自我对弈框架，用于开放式任务，它协同演化两个策略：一个生成文档基础任务的挑战者，以及一个通过多轮检索回答这些任务的求解者。初始模型的冻结副本作为自我评判者，它从源文档编写特定任务的评分标准，并据此对求解者的回答进行评分。在三个7-8B指令调优模型（Qwen2.5、Qwen3、OLMo-3）上，SCOPE在八个基准测试中将开放式任务性能提升了高达+10.4分，并匹配或超过了在约9K个精心设计的提示上训练的GRPO_data。尽管仅在开放式任务上训练，SCOPE还在七个保留的短格式问答基准测试中提升了高达+13.8分，在所有三个模型上均超越了GRPO_data。消融实验表明，协同演化挑战者对于保持任务接近求解者的前沿是必要的；性能提升源于检索和合成的改进，其相对贡献因任务而异；并且评分标准生成质量是自我评判的瓶颈。
英文摘要：
Self-play can train language models without external supervision. However, existing methods require rule-checkable answers, leaving open-ended tasks dependent on curated prompts or frontier-model judges. We introduce SCOPE, a data-free self-play framework for open-ended tasks that co-evolves two policies: a Challenger that generates document-grounded tasks, and a Solver that answers them through multi-turn retrieval. A frozen copy of the initial model serves as the self-judge, which writes task-specific rubrics from the source document and grades Solver responses against them. Across three 7-8B instruction-tuned models (Qwen2.5, Qwen3, OLMo-3), SCOPE improves open-ended performance by up to +10.4 points on eight benchmarks and matches or exceeds GRPO_data trained on ~9K curated prompts. Although trained only on open-ended tasks, SCOPE also improves held-out short-form QA by up to +13.8 points on seven held-out benchmarks, surpassing GRPO_data on all three models. Ablations show that co-evolving the Challenger is necessary to keep tasks near the Solver's frontier, that gains arise from improvements in both retrieval and synthesis with the relative contribution varying by task, and that rubric generation quality is the bottleneck for self-judging.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
