发表机构
Brackett Labs（Brackett 实验室）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
ShowTellArena提出一个基准协议和数据集，通过50个业务流程任务和502个问题，评估智能体从带叙述演示中理解业务规则的能力，并分析试点结果以暴露错误和教学体验失败。
AI 中文摘要
我们常常通过展示工作过程并边做边解释决策来教导同事。如何检查一个智能体从同样的课程中理解了什么？我们引入了ShowTellArena，一个用于带叙述的业务演示后理解评估的基准协议和公共数据集。v1.0版本包含50个业务流程任务，配有录音、截图、叙述、夹具种子和502个问题。任务涵盖财务、招聘、采购、客户决策、库存和物流。该协议保持业务场景和测验固定，同时允许每个产品通过其自身的教学界面来捕获课程内容。问题测试操作规则、边界、异常以及提议自动化中的错误。我们分析了39个流程案例中的218个选定的试点尝试，包括所有三个评估系统都尝试的28个案例。这些探索性结果既暴露了答案错误，也暴露了完成教学体验的失败。我们描述了该版本的验证缺口以及试点的覆盖不均、排除和评分来源。贡献在于一个可检查的数据集和评估工作流，其他人可以扩展；所选试点并非受控的产品排名。
英文摘要
We often teach a colleague by showing the work and explaining the decisions as we go. How can we check what an agent understood from the same lesson? We introduce ShowTellArena, a benchmark protocol and public dataset for comprehension after narrated business demonstrations. The v1.0 release contains 50 business workflow tasks, with recordings, screenshots, narration, fixture seeds, and 502 questions. Tasks span finance, hiring, procurement, customer decisions, inventory, and logistics. The protocol holds the business scenario and quiz fixed while allowing each product to capture the lesson through its own teaching interface. Questions test operational rules, boundaries, exceptions, and errors in proposed automations. We analyze 218 selected pilot attempts across 39 workflow cases, including 28 cases attempted by all three evaluated systems. These exploratory results expose both answer errors and failures to complete the teaching experience. We describe the release's verification gaps and the pilot's uneven coverage, exclusions, and grading provenance. The contribution is an inspectable dataset and assessment workflow that others can extend; the selected pilot is not a controlled product ranking.
Comments6 pages, 1 figure, 2 tables. Code and dataset linked in the paper
