通用智能体能否自动化数据筛选？
Can Generalist Agents Automate Data Curation?
AI总结：
本文提出Curation-Bench基准，通过通用编码智能体自动化数据筛选循环，实验表明现成智能体可达到强基线，但存在执行-研究差距，而结构化方法引导的智能体能在十分之一数据预算下自主组合出优于强基线的数据选择策略。
摘要：
训练数据的筛选是现代AI开发中最重要但劳动密集的部分之一：实践者根据嘈杂的基准反馈迭代地提出、实施、评估和修订数据策略。我们探究通用编码智能体能否自动化这一数据筛选循环。我们引入了*Curation-Bench*，一个以智能体为中心的基准，它固定模型、训练配方和评估套件，同时赋予智能体命令行权限以检查数据、实施策略、提交到固定的训练/评估流水线并进行修订。在视觉-语言指令微调实例中，现成智能体在十次迭代内达到了已发表的强数据选择基线。然而，轨迹分析揭示了持续的*执行-研究差距*：即使提供了策略指南和论文参考，智能体主要调整局部策略变体，而非探索新的策略家族。要求每次迭代引用、实例化和改编先前方法的框架将智能体转向方法引导的探索。这种框架化的智能体自主组合——无需人工设计输入——一种数据选择策略，在十分之一的数据预算下优于已发表的强基线。总体而言，当前智能体可以运行筛选循环，但可靠的数据研究需要框架化的方法适应，而非仅靠开放式提示。代码和基准已开源。
英文摘要：
Curating training data is among the most consequential yet labor-intensive parts of modern AI development: practitioners iteratively propose, implement, evaluate, and revise data policies against noisy benchmark feedback. We ask whether generalist coding agents can automate this data-curation loop. We introduce *Curation-Bench*, an agent-centric benchmark that fixes the model, training recipe, and evaluation suite while giving agents command-line access to inspect data, implement policies, submit them to a fixed training/evaluation pipeline, and revise. In a vision-language instruction-tuning instantiation, out-of-the-box agents reach strong published data-selection baselines within ten iterations. However, trajectory analysis reveals a persistent *execution-research gap*: agents mainly tune local policy variants rather than explore new policy families, even when given strategy guides and paper references. Scaffolds requiring each iteration to cite, instantiate, and adapt a prior method shift agents toward method-guided exploration. The scaffolded agent autonomously composes -- without human design input -- a data-selection policy that outperforms strong published baselines at one-tenth their data budget. Overall, current agents can run the curation loop, but reliable data research requires scaffolded method adaptation, not open-ended prompting alone. Code and benchmark are open-sourced.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
