SPOGW：一种面向工作流的基于分组比较的分数式偏好优化方法
SPOGW: a Score-based Preference Optimization method via Group-Wise comparison for workflows
AI总结：
针对现有智能体工作流优化方法依赖离散优化、存在表征与扩展性局限的问题，提出基于分组比较的分数式偏好优化方法SPOGW，在5个基准上性能持平或超越当前最优水平。
摘要：
大语言模型（LLMs）在解决各领域的挑战性问题上展现出显著能力，这通常借助遵循结构化指令与多步骤流程的智能体工作流实现。然而，设计此类工作流需要大量人工投入，给可扩展性与泛化性带来挑战。近期研究致力于减少工作流构建所需的人工干预，推动了智能体工作流自动化优化技术的发展。但现有方法往往受限于表征能力有限、适应性不足、可扩展性弱以及成对比较范式——这些问题主要源于对离散优化技术的依赖。为克服这些局限，我们提出一种新型基于分数的偏好方法，名为SPOGW，它通过分组比较直接基于基数奖励信号运行，可在连续空间中实现更高效、稳定的优化。SPOGW融合了带优势掩码KL散度（mKL）的迭代离线GRPO（ioGRPO），该机制通过更侧重策略响应的优势区域来调控训练更新。在涵盖数学推理、编程与问答的5个基准数据集上，SPOGW的性能与当前最优方法相当或更优，为智能体工作流的自动化生成与优化提供了一种可行且具前瞻性的方法。
英文摘要：
Large language models (LLMs) have exhibited significant capabilities in addressing challenging problems throughout various fields, often through the use of agentic workflows that adhere to structured instructions and multi-step procedures. However, designing such workflows demands substantial manual effort, posing challenges to scalability and generalizability. Recent studies have aimed to minimize the human intervention needed for their construction, leading to advances in automated techniques for optimizing agentic workflows. However, current approaches are often constrained by their limited representational capacity, insufficient adaptability, weak scalability, and pairwise comparison paradigm -- issues that stem primarily from a dependence on discrete optimization techniques. To overcome these limitations, we introduce a new score-based preference approach, refereed as SPOGW, which operates directly on cardinal reward signals through group-wise comparison and enables more efficient and stable optimization in a continuous space. SPOGW incorporates Iterative offline GRPO (ioGRPO) with advantage-masked KL divergence (mKL), which regulates training update by placing greater emphasis on the advantageous regions of the policy response. In five benchmark datasets covering mathematical reasoning, coding, and question answering, SPOGW matches or exceeds the performance of current state-of-the-art approaches, presenting a viable and forward-looking methodology for automated generation and optimization of agentic workflows.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
