WorkflowPerturb:用于评估多智能体工作流度量的校准压力测试
WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics
AI总结:
提出WorkflowPerturb基准,通过对黄金工作流施加分级扰动来评估多智能体工作流度量,揭示度量分数校准不良问题,支持变更管理中的严重性感知解释。
摘要:
从自然语言请求生成结构化工作流的多智能体LLM系统现已部署在云自动化、DevOps和企业流程编排的生产环境中。运行此类系统会暴露一个反复出现的变更管理问题。常规更新,例如重新运行相同的输入、替换底层LLM或重构智能体的提示或编排代码,经常产生与先前验证的参考工作流差异很大的工作流。工程师随后缺乏原则性的方法来决定变更是否安全发布。自动工作流评估是回答这个问题的自然工具。然而在实践中,度量分数校准不良,数值变化很少能传达底层降级的严重性。我们引入WorkflowPerturb,一个受控基准,通过向黄金工作流应用现实的分级扰动来研究工作流评估度量。WorkflowPerturb包含4,973个黄金工作流和44,757个扰动变体,涵盖三种扰动类型(缺失步骤、压缩步骤和描述更改),每种类型以10%、30%和50%的严重程度应用。我们对多个度量族进行基准测试,并使用期望分数轨迹和残差分析它们的敏感性和校准。我们的结果表征了不同度量族之间的系统性差异,并支持在变更管理环境中对工作流评估分数进行严重性感知解释。我们的数据集将在接收后发布。
英文摘要:
Multi-agent LLM systems that generate structured workflows from natural-language requests are now deployed in production across cloud automation, DevOps, and enterprise orchestration. Operating them exposes a recurring change-management problem. Routine updates, such as re-running an input, swapping the LLM, or refactoring an agent's prompt or orchestration code, often produce workflows that differ substantially from validated references. Engineers then lack a principled way to decide whether a change is safe to ship. Automatic workflow evaluation is the natural tool, but in practice metric scores are poorly calibrated, and a numeric change rarely communicates the severity of the underlying degradation. We introduce WorkflowPerturb, a controlled benchmark that applies realistic, graded perturbations to golden workflows. It contains 4,973 golden workflows and 44,757 perturbed variants across three perturbation types (Missing Steps, Compressed Steps, Description Changes) at severities of 10%, 30%, and 50%. We benchmark multiple metric families, analyzing their sensitivity and calibration using expected score trajectories and instance-level alert rates. Our results characterize systematic differences across families and support severity-aware interpretation of workflow evaluation scores in change-management settings. WorkflowPerturb is publicly available at https://huggingface.co/datasets/microsoft/WorkflowPerturb .
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。