DAComp: 在全数据智能生命周期中跨领域评估数据代理
DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle
AI总结：
DAComp通过210个任务评估数据代理在数据工程与分析中的能力，揭示现有代理在复杂流程编排和开放性推理中的不足。
摘要：
现实中的企业数据智能工作流程包括将原始数据源转换为分析-ready表格的数据工程，以及将这些表格转化为决策导向洞察的数据分析。我们引入DAComp，一个包含210个任务的基准测试，以反映这些复杂的工作流程。数据工程（DE）任务需要在工业模式上进行仓库级的工程，包括从零开始设计和构建多阶段SQL管道，并在需求变化下演进现有系统。数据分析（DA）任务则提出开放性商业问题，要求战略规划、通过迭代编码进行探索性分析、解释中间结果并合成可操作的建议。工程任务通过基于执行的、多指标评估进行评分。开放性任务则由一个经过实验验证的LLM评委评估，该评委由分层、精心设计的评分标准引导。我们的实验表明，即使是最先进的代理在DAComp上也表现不佳。在DE任务上的表现尤其低，成功率低于20%，暴露了整体流程编排中的关键瓶颈，而不仅仅是代码生成。在DA任务上的评分也平均低于40%，突显了开放性推理的深层缺陷，并表明工程和分析是不同的能力。通过明确诊断这些限制，DAComp提供了一个严格且现实的测试平台，以推动真正有能力的自主数据代理在企业环境中的发展。我们的数据和代码可在https://da-comp.github.io获取。
英文摘要：
Real-world enterprise data intelligence workflows encompass data engineering that turns raw sources into analytical-ready tables and data analysis that convert those tables into decision-oriented insights. We introduce DAComp, a benchmark of 210 tasks that mirrors these complex workflows. Data engineering (DE) tasks require repository-level engineering on industrial schemas, including designing and building multi-stage SQL pipelines from scratch and evolving existing systems under evolving requirements. Data analysis (DA) tasks pose open-ended business problems that demand strategic planning, exploratory analysis through iterative coding, interpretation of intermediate results, and the synthesis of actionable recommendations. Engineering tasks are scored through execution-based, multi-metric evaluation. Open-ended tasks are assessed by a reliable, experimentally validated LLM-judge, which is guided by hierarchical, meticulously crafted rubrics. Our experiments reveal that even state-of-the-art agents falter on DAComp. Performance on DE tasks is particularly low, with success rates under 20%, exposing a critical bottleneck in holistic pipeline orchestration, not merely code generation. Scores on DA tasks also average below 40%, highlighting profound deficiencies in open-ended reasoning and demonstrating that engineering and analysis are distinct capabilities. By clearly diagnosing these limitations, DAComp provides a rigorous and realistic testbed to drive the development of truly capable autonomous data agents for enterprise settings. Our data and code are available at https://da-comp.github.io
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
