LH-Bench：面向主观企业任务的长期智能体技能基础评估
LH-Bench: Skill-Grounded Evaluation of Long-Horizon Agents on Subjective Enterprise Tasks
AI总结：
提出LH-Bench，通过专家基础评分标准、真实标注工件和成对偏好评估三支柱，解决主观企业任务中长期自主执行的评估问题。
摘要：
大型语言模型在数学和编程等客观可验证任务上表现出色，这些任务的评估简化为单元测试或单一正确答案。相比之下，现实世界中的企业工作通常是主观且依赖上下文的：成功取决于组织目标、用户意图以及在长期多工具工作流中产生的中间工件的质量。我们引入LH-Bench，一种三支柱评估设计，超越二元正确性，对主观企业任务中的自主长期执行进行评分。这些支柱包括：(i) 专家基础评分标准，为LLM评判者提供评估主观工作所需的领域背景；(ii) 策划的真实工件，提供逐步奖励信号（例如，内容任务的章节级注释）；以及(iii) 成对人类偏好评估，用于收敛验证。我们表明，领域作者编写的评分标准比LLM作者编写的评分标准提供更可靠的评估信号（kappa = 0.60 vs. 0.46），并且人类偏好判断确认了相同的顶级分离（p < 0.05），这证明专家基础评估可以在不牺牲可靠性的情况下扩展。我们发布公共数据集，并报告两个环境的结果：Figma到代码（通过MCP针对Figma API的33个真实.fig任务）和程序化内容（41门课程，包含183个单独评估的章节，服务于一个拥有30+日常用户的课程平台）。
英文摘要：
Large language models excel on objectively verifiable tasks such as math and programming, where evaluation reduces to unit tests or a single correct answer. In contrast, real-world enterprise work is often subjective and context-dependent: success hinges on organizational goals, user intent, and the quality of intermediate artifacts produced across long, multi-tool workflows. We introduce LH-Bench, a three-pillar evaluation design that moves beyond binary correctness to score autonomous, long-horizon execution on subjective enterprise tasks. The pillars are: (i) expert-grounded rubrics that give LLM judges the domain context needed to score subjective work, (ii) curated ground-truth artifacts that enable stepwise reward signals (e.g., chapter-level annotation for content tasks), and (iii) pairwise human preference evaluation for convergent validation. We show that domain-authored rubrics provide substantially more reliable evaluation signals than LLM-authored rubrics (kappa = 0.60 vs. 0.46), and that human preference judgments confirm the same top-tier separation (p < 0.05), evidence that expert-grounded evaluation can scale without sacrificing reliability. We release public datasets and report results on two environments: Figma-to-code (33 real .fig tasks against the Figma API via MCP) and Programmatic content (41 courses comprising 183 individually-evaluated chapters on a course platform serving 30+ daily users).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
