arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.09336cs.CL
收藏

FM SO.P: 一种具有自动评估的渐进式任务混合框架用于跨域SOP理解

FM SO.P: A Progressive Task Mixture Framework with Automatic Evaluation for Cross-Domain SOP Understanding

Amazon(亚马逊);Johns Hopkins University(约翰霍普金斯大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Siyuan Huang, Ziyu Wang, Chao Pan, Han Zhao

AI总结:

FM SO.P提出了一种渐进式任务混合框架和自动评估系统,以提升跨领域SOP理解的准确性和泛化能力。

摘要:

标准操作程序(SOP)对企业运营至关重要,但现有语言模型在SOP理解和跨领域泛化方面存在困难。当前方法失败的原因是联合训练无法区分SOP所需的原因能力:术语精度、顺序排列和约束推理。我们提出了FM SO.P,通过两个创新来解决这些挑战。首先,我们引入了渐进式任务混合,通过三种任务类型分阶段构建能力,累积数据:术语歧义消除用于术语精度,动作序列理解用于过程正确性,以及场景感知图推理用于条件逻辑。第二,我们提出了一个自动多代理评估系统,由三个代理组成,能够自适应生成评分标准、分层测试集和评分标准,适应不同领域(例如,时间约束用于DMV,监管合规用于银行)。在SOPBench上七个领域(银行、DMV、医疗、市场、大学、图书馆、酒店)上评估,FM SO.P在我们的32B模型上实现了48.3%的通过率,在我们的开源7B模型上实现了34.3%的通过率,与Qwen-2.5-72B-Instruct基准(34.4%)相匹配,参数量仅为10倍少。

英文摘要:

Standard Operating Procedures (SOPs) are critical for enterprise operations, yet existing language models struggle with SOP understanding and cross-domain generalization. Current methods fail because joint training cannot differentiate between reasoning capabilities that SOP requires: terminology precision, sequential ordering, and constraint reasoning. We propose FM SO.P, solving these challenges through two novelties. First, we introduce progressive task mixtures that build capabilities by stages across three task types with cumulative data: concept disambiguation for terminology precision, action sequence understanding for procedural correctness, and scenario-aware graph reasoning for conditional logic. Second, we propose an automatic multi-agent evaluation system consisting of three agents that adaptively generate rubrics, stratified test sets, and rubric scoring, adapting to domains (e.g., temporal constraints for DMV, regulatory compliance for banking). Evaluated on SOPBench across seven domains (Bank, DMV, Healthcare, Market, University, Library, Hotel), FM SO.P achieves 48.3\% pass rate with our 32B model and 34.3\% with our opensource 7B model, matching Qwen-2.5-72B-Instruct baseline (34.4\%) with 10x fewer parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑