UpBench: 一个动态演化的现实劳动力市场代理基准框架，专为以人为本的AI设计
UpBench: A Dynamically Evolving Real-World Labor-Market Agentic Benchmark Framework Built for Human-Centric AI
AI总结：
UpBench是一个动态演化的现实劳动力市场代理基准框架，通过真实工作和财务结果评估AI与人类协作的能力，推动人机协作的发展。
摘要：
随着大型语言模型（LLM）代理越来越多地参与数字工作，需要可靠的框架来评估其在现实世界中的能力、适应性和与人类协作的能力。现有的基准测试大多静态、合成或领域受限，难以提供关于代理在动态、具有经济意义的环境中表现的深入见解。我们介绍了UpBench，一个动态演化的基准测试，基于全球Upwork劳动力市场中的真实工作。每个任务对应一个已验证的客户交易，将评估扎根于真实的劳动活动和财务结果。UpBench采用基于评分标准的评估框架，专家自由职业者将每个任务分解成详细、可验证的接受标准，并对AI提交物进行每项标准的反馈评估。这种结构使能够对模型的优缺点和指令遵循的忠实度进行细致分析，超越二元通过/失败指标。人类专业知识贯穿整个数据管道（从任务筛选和评分标准构建到评估），确保符合真实的专业标准，并支持对人机协作的研究。通过定期刷新任务以反映在线工作的演变性质，UpBench提供了一个可扩展、以人为本的基础，用于在真实的劳动力市场背景下评估代理系统，提供了一个迈向协作框架的道路，其中AI通过合作而非替代来增强人类能力。
英文摘要：
As large language model (LLM) agents increasingly undertake digital work, reliable frameworks are needed to evaluate their real-world competence, adaptability, and capacity for human collaboration. Existing benchmarks remain largely static, synthetic, or domain-limited, providing limited insight into how agents perform in dynamic, economically meaningful environments. We introduce UpBench, a dynamically evolving benchmark grounded in real jobs drawn from the global Upwork labor marketplace. Each task corresponds to a verified client transaction, anchoring evaluation in genuine work activity and financial outcomes. UpBench employs a rubric-based evaluation framework, in which expert freelancers decompose each job into detailed, verifiable acceptance criteria and assess AI submissions with per-criterion feedback. This structure enables fine-grained analysis of model strengths, weaknesses, and instruction-following fidelity beyond binary pass/fail metrics. Human expertise is integrated throughout the data pipeline (from job curation and rubric construction to evaluation) ensuring fidelity to real professional standards and supporting research on human-AI collaboration. By regularly refreshing tasks to reflect the evolving nature of online work, UpBench provides a scalable, human-centered foundation for evaluating agentic systems in authentic labor-market contexts, offering a path toward a collaborative framework, where AI amplifies human capability through partnership rather than replacement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
