arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.23331cs.CL
收藏

模型能帮助我们创建更好的模型吗?评估作为数据科学家的大语言模型

Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientists

Snowflake AI Research(Snowflake AI 研究院);Polish Academy of Sciences(波兰科学院)

Michał Pietruszka, Łukasz Borchmann, Aleksander Jędrosz, Paweł Morawiecki

AI总结:

该研究提出名为FeatEng的LLM基准,通过让模型生成特征工程代码、以修改后数据训练的XGBoost的性能提升为评分标准,可低成本高效评估LLM的数据科学综合能力。

摘要:

我们提出了一个面向大语言模型(LLM)的基准,用于应对数据科学中知识密集度最高的任务之一:编写特征工程代码——该任务除了需要对底层问题和数据结构有深刻理解外,还要求具备领域知识。我们在提示词中向模型提供数据集描述,并要求其生成对数据集进行转换的代码。评估分数由在修改后的数据集上训练的XGBoost模型相较于在原始数据上训练的模型所取得的性能提升得出。通过对当前最优模型的广泛评估以及与成熟基准的对比,我们证明,与现有方法相比,我们提出的FeatEng基准能够低成本、高效地评估LLM的综合能力。

英文摘要:

We present a benchmark for large language models designed to tackle one of the most knowledge-intensive tasks in data science: writing feature engineering code, which requires domain knowledge in addition to a deep understanding of the underlying problem and data structure. The model is provided with a dataset description in a prompt and asked to generate code transforming it. The evaluation score is derived from the improvement achieved by an XGBoost model fit on the modified dataset compared to the original data. By an extensive evaluation of state-of-the-art models and comparison to well-established benchmarks, we demonstrate that the FeatEng of our proposal can cheaply and efficiently assess the broad capabilities of LLMs, in contrast to the existing methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑