arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.24288cs.AIcs.CL
收藏

DARE-bench: 评估LLMs在数据科学中的建模和指令忠实度

DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Science

University of Houston(德克萨斯大学休斯顿分校);Snowflake AI Research(Snowflake人工智能研究院)

Fan Shu, Yite Wang, Ruofan Wu, Boyi Liu, Zhewei Yao, Yuxiong He, Feng Yan

AI总结:

DARE-bench通过提供可验证的真实值任务和大规模训练数据,有效评估和提升LLMs在数据科学中的建模和指令忠实度。

摘要:

随着使用大语言模型(LLMs)解决复杂多步骤数据科学任务的需求迅速增长,出现了对准确基准测试的迫切需求。现有基准测试存在两个主要缺口:(i)缺乏标准化、过程感知的评估,能够捕捉指令遵守和过程忠实度,以及(ii)准确标记的训练数据稀缺。为弥合这些缺口,我们引入了DARE-bench,一个用于机器学习建模和数据科学指令遵循的基准测试。与许多现有基准测试依赖于人类或模型评判不同,DARE-bench中的所有任务都有可验证的真实值,确保了客观且可重复的评估。为了覆盖广泛的任务并支持代理工具,DARE-bench包含6300个Kaggle衍生任务,并提供大规模训练数据和评估集。广泛评估显示,即使是非常强大的模型如gpt-o4-mini在机器学习建模任务中也难以取得良好表现。使用DARE-bench训练任务进行微调可以显著提高模型性能。例如,监督微调使Qwen3-32B的准确性提高了1.83倍,强化学习使Qwen3-4B的准确性提高了8倍以上。这些显著的改进验证了DARE-bench作为准确评估基准和关键训练数据的重要性。

英文摘要:

The fast-growing demands in using Large Language Models (LLMs) to tackle complex multi-step data science tasks create an emergent need for accurate benchmarking. There are two major gaps in existing benchmarks: (i) the lack of standardized, process-aware evaluation that captures instruction adherence and process fidelity, and (ii) the scarcity of accurately labeled training data. To bridge these gaps, we introduce DARE-bench, a benchmark designed for machine learning modeling and data science instruction following. Unlike many existing benchmarks that rely on human- or model-based judges, all tasks in DARE-bench have verifiable ground truth, ensuring objective and reproducible evaluation. To cover a broad range of tasks and support agentic tools, DARE-bench consists of 6,300 Kaggle-derived tasks and provides both large-scale training data and evaluation sets. Extensive evaluations show that even highly capable models such as gpt-o4-mini struggle to achieve good performance, especially in machine learning modeling tasks. Using DARE-bench training tasks for fine-tuning can substantially improve model performance. For example, supervised fine-tuning boosts Qwen3-32B's accuracy by 1.83x and reinforcement learning boosts Qwen3-4B's accuracy by more than 8x. These significant improvements verify the importance of DARE-bench both as an accurate evaluation benchmark and critical training data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑