arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.03255cs.AIcs.CL
收藏

面向业务流程管理任务的大型语言模型基准测试

Towards a Benchmark for Large Language Models for Business Process Management Tasks

Kühne Logistics University(库恩物流大学)

Kiran Busch, Henrik Leopold

AI总结:

本文针对业务流程管理(BPM)领域缺乏LLM性能基准的问题,系统比较了小型开源模型在四个BPM任务上的表现,评估了任务差异、开源与商业模型对比及模型规模影响,为组织选择合适模型提供指导。

摘要:

越来越多的组织正在部署大型语言模型(LLMs)以处理广泛的任务。尽管具有普遍实用性,LLMs 仍容易出现错误,从不准确到幻觉。为了客观评估现有 LLMs 的能力,人们进行了性能基准测试。然而,这些基准测试往往无法转化为更具体的实际任务。本文解决了在业务流程管理(BPM)领域对 LLMs 性能进行基准测试的空白。目前尚不存在针对 BPM 的特定基准测试,导致对不同 LLMs 在 BPM 任务中的适用性存在不确定性。本文系统比较了 LLMs 在四个 BPM 任务上的性能,重点关注小型开源模型。该分析旨在识别特定任务的性能差异,比较开源模型与商业模型的有效性,并评估模型规模对 BPM 任务性能的影响。本文提供了关于 LLMs 在 BPM 中实际应用的见解,指导组织根据其特定需求选择合适的模型。

英文摘要:

An increasing number of organizations are deploying Large Language Models (LLMs) for a wide range of tasks. Despite their general utility, LLMs are prone to errors, ranging from inaccuracies to hallucinations. To objectively assess the capabilities of existing LLMs, performance benchmarks are conducted. However, these benchmarks often do not translate to more specific real-world tasks. This paper addresses the gap in benchmarking LLM performance in the Business Process Management (BPM) domain. Currently, no BPM-specific benchmarks exist, creating uncertainty about the suitability of different LLMs for BPM tasks. This paper systematically compares LLM performance on four BPM tasks focusing on small open-source models. The analysis aims to identify task-specific performance variations, compare the effectiveness of open-source versus commercial models, and assess the impact of model size on BPM task performance. This paper provides insights into the practical applications of LLMs in BPM, guiding organizations in selecting appropriate models for their specific needs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑