Ambig-DS:数据科学代理任务框架模糊度的基准测试
Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents
AI总结:
Ambig-DS引入两个诊断套件,用于预测目标模糊度和评估目标模糊度,通过控制编辑生成模糊变体,发现任务框架模糊度是数据科学代理评估的瓶颈。
摘要:
随着数据科学代理从辅助工具转向自动控制,隐性任务框架错误成为关键失败模式。代理会默默承诺合理但不恰当的任务框架,产生干净可执行的成果,隐藏其对任务的错误评估。现有基准仅评分流程是否运行,忽略代理是否意识到任务未明确说明。我们引入Ambig-DS,两个诊断套件:一个用于预测目标模糊度(Ambig-DS-Target,51个基于DSBench的表格建模基准任务)和一个用于评估目标模糊度(Ambig-DS-Objective,61个基于MLE-bench的Kaggle风格机器学习竞赛基准任务),构造方式使评分使用每个源基准的原始评估者。对于每个任务,我们配对原始完全明确版本与通过受控编辑生成的模糊变体;人类和LLM验证流程确认每个变体允许多种合理解释,具有决策相关后果。套件独立分析,模糊度在两者中都降低性能。在五个涵盖高效到前沿模型的代理中,我们发现:(i)失败是沉默承诺:在Target中错误目标提交,在Objective中错误指标或不承诺的基线提交,而非执行错误;(ii)允许代理问一个澄清问题在理想条件下可恢复大部分损失,表明缺失的框架信息驱动了观察到的退化;但(iii)代理无法可靠地判断何时使用它:宽松提示在清晰任务中导致过度提问,而保守提示在模糊任务中导致沉默默认。识别目标和评估目标未明确说明,而非流程执行,是标准数据科学代理评估中缺失的瓶颈。
英文摘要:
As data-science agents shift from co-pilots to auto-pilots, silent misframing becomes a critical failure mode. Agents quietly commit to plausible but unintended task framings, producing clean, executable artifacts that hide their incorrect assessment of the task. Existing benchmarks score whether the pipeline runs, ignoring whether the agent recognized the task was underspecified. We introduce Ambig-DS, two diagnostic suites: one for prediction-target ambiguity (Ambig-DS-Target, 51 tasks built on DSBench, a tabular modeling benchmark) and one for evaluation-objective ambiguity (Ambig-DS-Objective, 61 tasks built on MLE-bench, a Kaggle-style ML competition benchmark), constructed so that scoring uses each source benchmark's original evaluator. For every task we pair the original, fully specified version with an ambiguous variant produced by controlled edits; a human-and-LLM verification pipeline confirms each variant admits multiple plausible interpretations with decision-relevant consequences. The suites are analyzed independently and ambiguity lowers performance in both. Across five agents spanning efficient to frontier-class models, we find in our controlled diagnostic setting: (i) failures are silent commitments: wrong-target submissions on Target, wrong-metric or non-committal baseline submissions on Objective, rather than execution errors; (ii) allowing the agent to ask one clarifying question recovers much of the loss under idealized conditions, suggesting missing framing information drives a substantial part of the observed degradation; but (iii) agents cannot reliably tell when to use it: permissive prompts induce over-asking on clear tasks, while conservative prompts induce silent defaulting on ambiguous ones. Recognizing target and objective underspecification, not pipeline execution, is the bottleneck missing from standard DS-agent evaluations.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。