arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.26329cs.AI
收藏

JobBench:使智能体工作符合人类意愿

JobBench: Aligning Agent Work With Human Will

University of Washington(华盛顿大学);University of California, Santa Barbara(加州大学圣芭芭拉分校);Stanford University(斯坦福大学);Carnegie Mellon University(卡内基梅隆大学);Northwestern University(西北大学);University of Notre Dame(圣母大学);University of California, Berkeley(加州大学伯克利分校);Michigan State University(密歇根州立大学);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室);Bake AI;King Abdulaziz City for Science and Technology(国王阿卜杜勒阿齐兹科技城);Western Washington University(西雅图华盛顿大学);University of Chicago(芝加哥大学)
Yuetai Li, Yichen Feng, Zhangchen Xu, Zixian Ma, Kaiyuan Zheng, Fengqing Jiang, Xinghua Sun, Rulin Shao, Zichen Chen, Yue Huang, Xinyang Han, Brian Lee, Kayla X… 展开作者

Yuetai Li, Yichen Feng, Zhangchen Xu, Zixian Ma, Kaiyuan Zheng, Fengqing Jiang, Xinghua Sun, Rulin Shao, Zichen Chen, Yue Huang, Xinyang Han, Brian Lee, Kayla Xu, Shenglai Zeng, Hang Hua, Xiangliang Zhang, Basel Alomair, Ranjay Krishna, Luke Zettlemoyer, Pang Wei Koh, Bhaskar Ramasubramanian, Luyao Niu, Xiang Yue, Radha Poovendran

AI总结:

提出JobBench基准,通过专家识别的高优先级工作流程评估AI智能体,以人类需求为中心而非经济价值,覆盖35个职业的130个任务,使用事实锚定的评分链评估,最强模型仅达45.9%,旨在推动从替代到增强的劳动力市场影响。

摘要:

当前职业AI智能体的基准主要基于经济价值,讲述了一个替代的故事。我们引入了JobBench,该基准根据专家识别为高优先级委托的工作流程评估AI智能体,基于人类需求赋权,而非用GDP价值替代他们。JobBench覆盖了35个职业的130个智能体任务。每个任务被打包成一个包含异构参考文件的工作空间,要求智能体在真实专业工作的杂乱信息流中进行推理。输出由事实锚定的评分链进行评分,每个任务平均有35.6个二元标准。我们评估了36个模型;最强的Claude Opus~4.7在Claude Code下仅达到45.9%。我们希望JobBench将社区的目标劳动力市场影响从替代转向增强:构建能够完成人类真正希望委托的任务的智能体,而不仅仅是经济价值最高的任务。

英文摘要:

Current benchmarks for occupational AI agents are scoped primarily by economic values, telling a replacement story. We introduce JobBench, which evaluates AI agents on the workflows that experts identify as high-priority for delegation, empowering humans based on their needs instead of replacing them with GDP value. JobBench covers 130 agentic tasks across 35 occupations. Each task is packaged as a workspace of heterogeneous reference files, requiring the agent to reason through the cluttered information streams of real professional work. Outputs are graded by a fact-anchored chain of rubrics, averaging 35.6 binary criteria per task. We evaluate 36 models; the strongest, Claude Opus~4.7 under Claude Code, reaches only 45.9 %. We hope JobBench shifts the community's target labour-market effect from replacement to enhancement: building agents that do what humans actually want delegated, not only what is most economically valuable.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑