We present a benchmark for large language models designed to tackle one of the most knowledge-intensive tasks in data science: writing feature engineering code, which requires domain knowledge in addition to a deep understanding of the underlying problem and data structure. The model is provided with a dataset description in a prompt and asked to generate code transforming it. The evaluation score is derived from the improvement achieved by an XGBoost model fit on the modified dataset compared to the original data. By an extensive evaluation of state-of-the-art models and comparison to well-established benchmarks, we demonstrate that the FeatEng of our proposal can cheaply and efficiently assess the broad capabilities of LLMs, in contrast to the existing methods.
DataAgent: Evaluating Large Language Models' Ability to Answer Zero-Shot, Natural Language Queries
Irvington High School(欧文顿高中);Pioneer High School(先锋高中);Davis Senior High School(戴维斯高级中学);The Loomis Chaffee School(卢米斯查菲学校);Bellarmine College Preparatory(贝拉明学院预备学校);Rutgers University(罗格斯大学);University of Maryland, College Park(马里兰大学学院市分校)
CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists
Tsinghua University(清华大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Carnegie Mellon University(卡内基梅隆大学);University of Chicago(芝加哥大学);Adobe
CausalGame: Benchmarking Causal Thinking of LLM Agents in Games
MBZUAI(穆罕默德·本·扎耶德人工智能大学);Carnegie Mellon University(卡内基梅隆大学);Hong Kong Baptist University(香港浸会大学);University of Oxford(牛津大学);New York University, Abu Dhabi(纽约大学阿布扎比分校)
Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systems
Department of Computer Science Princeton University(计算机科学系普林斯顿大学);Department of Computer Science Department of Psychology Princeton University(计算机科学系心理学系普林斯顿大学)
Using Natural Language Processing and Qualitative Analysis to Intervene in Gang Violence: A Collaboration Between Social Work Researchers and Data Scientists
Columbia University(哥伦比亚大学);Fairfield University(费尔菲尔德大学)
SciCode: A Research Coding Benchmark Curated by Scientists
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Massachusetts Institute of Technology(麻省理工学院);Harvard University(哈佛大学);University of Chicago(芝加哥大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);Stanford University(斯坦福大学)