WONDERBREAD：在业务流程管理任务上评估多模态基础模型的基准
WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business Process Management Tasks
AI总结：
提出首个超越自动化的业务流程管理（BPM）多模态基础模型评估基准WONDERBREAD，包含2928个工作流演示和6个真实任务，发现现有模型虽能生成文档但难以完成细粒度验证。
摘要：
现有的机器学习基准缺乏在业务流程管理（BPM）任务上评估模型所需的标注深度和多样性。BPM是记录、测量、改进和自动化企业工作流的实践。然而，研究几乎完全集中在一个任务上——使用基于GPT-4等多模态基础模型（FMs）的智能体进行完整的端到端自动化。这种对自动化的关注忽略了当今大多数BPM工具应用的现实情况——仅仅记录相关工作流就需要花费典型流程优化项目60%的时间。为了解决这一差距，我们提出了WONDERBREAD，这是首个在超越自动化的BPM任务上评估多模态FMs的基准。我们的贡献包括：（1）一个包含2928个已记录工作流演示的数据集；（2）源自真实世界应用的6个新颖BPM任务，涵盖从工作流记录到知识转移再到流程改进；（3）一个自动化评估测试套件。我们的基准表明，尽管最先进的FMs能够自动生成文档（例如，回忆起工作流视频演示中88%的步骤），但它们难以将这些知识重新应用于更细粒度的工作流完成验证（F1 < 0.3）。我们希望WONDERBREAD能鼓励为企业应用开发更多“以人为本”的AI工具，并促进对多模态FMs在更广泛的BPM任务领域的探索。
英文摘要：
Existing ML benchmarks lack the depth and diversity of annotations needed for evaluating models on business process management (BPM) tasks. BPM is the practice of documenting, measuring, improving, and automating enterprise workflows. However, research has focused almost exclusively on one task - full end-to-end automation using agents based on multimodal foundation models (FMs) like GPT-4. This focus on automation ignores the reality of how most BPM tools are applied today - simply documenting the relevant workflow takes 60% of the time of the typical process optimization project. To address this gap we present WONDERBREAD, the first benchmark for evaluating multimodal FMs on BPM tasks beyond automation. Our contributions are: (1) a dataset containing 2928 documented workflow demonstrations; (2) 6 novel BPM tasks sourced from real-world applications ranging from workflow documentation to knowledge transfer to process improvement; and (3) an automated evaluation harness. Our benchmark shows that while state-of-the-art FMs can automatically generate documentation (e.g. recalling 88% of the steps taken in a video demonstration of a workflow), they struggle to re-apply that knowledge towards finer-grained validation of workflow completion (F1 < 0.3). We hope WONDERBREAD encourages the development of more "human-centered" AI tooling for enterprise applications and furthers the exploration of multimodal FMs for the broader universe of BPM tasks. We publish our dataset and experiments here: https://github.com/HazyResearch/wonderbread
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
