ELF-Gym：评估大语言模型为表格预测生成的特征
ELF-Gym: Evaluating Large Language Models Generated Features for Tabular Prediction
AI总结：
本文提出ELF-Gym框架，通过构建包含251个Kaggle黄金特征的数据集，从下游性能和语义/功能相似性两方面定量评估LLM生成特征与人类专家的差距，并实证揭示了LLM在复杂特征构建上的显著不足。
摘要：
在机器学习流程中，构建有效特征是一项至关重要但劳动密集且高度依赖领域知识的工作。幸运的是，大语言模型（LLMs）的最新进展已在自动化多种数据科学任务方面展现出潜力，包括特征工程。然而，尽管具备这种潜力，迄今为止的评估主要基于完整机器学习流程的最终性能，对LLMs在特征工程中相对于人类专家的具体行为提供的洞察十分有限。为弥补这一不足，我们提出了ELF-Gym，一个用于评估LLM生成特征的框架。我们从历史Kaggle竞赛中整理了一个新数据集，其中包含顶尖团队使用的251个“黄金”特征。ELF-Gym通过测量LLM生成特征对下游模型性能的影响，以及通过语义和功能相似性评估其与专家构建特征的一致性，对其进行定量评估。该方法能够更全面地评估LLMs与人类专家之间的差异，同时为LLMs可能存在的改进空间提供有价值的见解。例如，利用ELF-Gym，我们通过实验证明，在最佳情况下，LLMs能够在语义上捕获约56%的黄金特征，但在要求更高的实现层面，这一重叠降至13%。此外，在其他情况下，LLMs可能完全失败，尤其是在需要复杂特征的数据集上，这表明存在广泛的潜在改进路径。
英文摘要：
Crafting effective features is a crucial yet labor-intensive and domain-specific task within machine learning pipelines. Fortunately, recent advancements in Large Language Models (LLMs) have shown promise in automating various data science tasks, including feature engineering. But despite this potential, evaluations thus far are primarily based on the end performance of a complete ML pipeline, providing limited insight into precisely how LLMs behave relative to human experts in feature engineering. To address this gap, we propose ELF-Gym, a framework for Evaluating LLM-generated Features. We curated a new dataset from historical Kaggle competitions, including 251 "golden" features used by top-performing teams. ELF-Gym then quantitatively evaluates LLM-generated features by measuring their impact on downstream model performance as well as their alignment with expert-crafted features through semantic and functional similarity assessments. This approach provides a more comprehensive evaluation of disparities between LLMs and human experts, while offering valuable insights into specific areas where LLMs may have room for improvement. For example, using ELF-Gym we empirically demonstrate that, in the best-case scenario, LLMs can semantically capture approximately 56% of the golden features, but at the more demanding implementation level this overlap drops to 13%. Moreover, in other cases LLMs may fail completely, particularly on datasets that require complex features, indicating broad potential pathways for improvement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
