ARES: 面向可扩展大语言模型强化学习的自动评分标准合成
ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learning
AI总结：
提出ARES框架，从原始预训练文档自动生成问答对和实例级加权评分标准，用于可扩展的基于评分标准的强化学习，在多个开放任务上超越持续预训练、监督微调和二元奖励强化学习。
摘要：
基于评分标准的奖励为将强化学习扩展到大型语言模型提供了一种有前景的方式，超越了具有自动可验证答案的任务。然而，扩展基于评分标准的强化学习仍然具有挑战性：现有方法通常依赖专家编写的评分标准和手动构建的问题集，而固定的任务级评分标准可能无法捕捉单个问题的评估需求。我们提出ARES（面向可扩展强化学习的自动评分标准合成），一个自动构建基于评分标准的强化学习数据的框架。从原始预训练文档开始，ARES将源知识转换为自包含的问答对，并共同生成特定问题的加权评分标准，从而为开放式回答提供实例级奖励监督。为了提高多样性和质量，ARES基于领域标签和人物角色信息生成，并应用验证过滤器以确保问题自包含性、答案忠实性和评分标准有效性。使用ARES，我们在十个领域构建了10万个评分标准标注的实例。在七个基准上的实验表明，使用ARES训练的基于评分标准的强化学习优于持续预训练、监督微调和二元奖励强化学习，在医疗和指令遵循等多维开放任务上提升最大。
英文摘要：
Rubric-based rewards offer a promising way to extend reinforcement learning (RL) for large language models beyond tasks with automatically verifiable answers. However, scaling rubric-based RL remains challenging: existing approaches often rely on expert-written rubrics and manually constructed question sets, while fixed task-level rubrics may fail to capture the evaluation requirements of individual questions. We propose ARES (Automated Rubric synthEsis for Scalable RL), a framework for automatically constructing rubric-based RL data at scale. Starting from raw pretraining documents, ARES converts source knowledge into self-contained question-answer pairs and co-generates question-specific weighted rubrics, enabling instance-level reward supervision for open-ended responses. To improve diversity and quality, ARES conditions generation on domain labels and persona information, and applies validation filters for question self-containment, answer faithfulness, and rubric validity. Using ARES, we construct 100K rubric-annotated instances across ten domains. Experiments on seven benchmarks show that rubric-based RL trained with ARES, outperforms continual pretraining, supervised fine-tuning, and binary-reward RL, with the largest gains on multi-dimensional open-ended tasks such as healthcare and instruction following.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
