arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2509.09321cs.AI
收藏

迈向自适应机器学习基准:Web智能体驱动的构建、领域扩展与指标优化

Towards Adaptive ML Benchmarks: Web-Agent-Driven Construction, Domain Expansion, and Metric Optimization

Hangyi Jia, Yuxi Qian, Hanwen Tong, Xinhui Wu, Lin Chen, Feng Wei

AI总结:

针对现有ML基准在覆盖度、多样性和评估严谨性上的不足,本文提出TAM Bench,通过Web智能体自动采集任务、排行榜驱动难度建模及多维评估框架,构建了Lite、Medium、Full三个规模的基准子集,以全面评估LLM智能体的端到端ML能力。

摘要:

大型语言模型(LLM)的最新进展催生了通用智能体的出现,这些智能体能够自动化端到端的机器学习(ML)工作流,包括数据分析、特征工程、模型训练和竞赛求解。然而,现有基准在任务覆盖范围、领域多样性、难度建模和评估严谨性方面仍然有限,无法在现实场景中充分捕捉此类智能体的全部能力。我们提出了TAM Bench,一个多样化、现实且结构化的基准,用于评估基于LLM的智能体在端到端机器学习任务上的表现。TAM Bench具有三项关键创新:(1)一种基于浏览器自动化和LLM的任务采集系统,能够从Kaggle、AIcrowd和Biendata等平台自动收集并结构化机器学习挑战,涵盖多种任务类型和数据模态(如表格、文本、图像、图、音频);(2)一种基于排行榜的难度建模机制,利用参赛者数量和分数离散度来估计任务复杂度,实现可扩展且客观的任务校准;(3)一个多维评估框架,整合了性能、格式合规性、约束遵循性和任务泛化性。基于150个精选的AutoML任务,我们构建了三个不同规模的基准子集——Lite、Medium和Full——用于不同的评估场景。其中Lite版本包含18个任务,在模态和难度级别上覆盖均衡,可作为日常基准测试和比较研究的实用测试平台。

英文摘要:

Recent advances in large language models (LLMs) have enabled the emergence of general-purpose agents for automating end-to-end machine learning (ML) workflows, including data analysis, feature engineering, model training, and competition solving. However, existing benchmarks remain limited in task coverage, domain diversity, difficulty modeling, and evaluation rigor, failing to capture the full capabilities of such agents in realistic settings. We present TAM Bench, a diverse, realistic, and structured benchmark for evaluating LLM-based agents on end-to-end ML tasks. TAM Bench features three key innovations: (1) A browser automation and LLM-based task acquisition system that automatically collects and structures ML challenges from platforms such as Kaggle, AIcrowd, and Biendata, spanning multiple task types and data modalities (e.g., tabular, text, image, graph, audio); (2) A leaderboard-driven difficulty modeling mechanism that estimates task complexity using participant counts and score dispersion, enabling scalable and objective task calibration; (3) A multi-dimensional evaluation framework incorporating performance, format compliance, constraint adherence, and task generalization. Based on 150 curated AutoML tasks, we construct three benchmark subsets of different sizes -- Lite, Medium, and Full -- designed for varying evaluation scenarios. The Lite version, with 18 tasks and balanced coverage across modalities and difficulty levels, serves as a practical testbed for daily benchmarking and comparative studies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑