arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.21155cs.LG
收藏

将LLM中的领域特定知识嵌入特征工程流水线

Embedding Domain-Specific Knowledge from LLMs into the Feature Engineering Pipeline

RIKEN(理化学研究所)

João Eduardo Batista

AI总结:

本研究提出利用LLM进行初始特征构造,以加速进化特征工程并节省计算资源,同时引入M6GP算法提升符号回归性能。

摘要:

特征工程在机器学习流水线中是强制性的,以获得稳健的模型。虽然进化计算在特征选择和特征构造方面以其出色的结果而闻名,但其方法由于需要大量评估来诱导最终模型而计算成本高昂。这些算法需要大量评估的部分原因是缺乏领域特定知识,导致在进化过程中存在大量随机猜测。在这项工作中,我们提出使用大型语言模型(LLMs)作为初始特征构造步骤,以向数据集添加知识。通过这样做,我们的结果表明进化可以更快地收敛,节省计算资源。所提出的方法仅向LLM提供数据集中特征的名称和目标目标,使其即使在使用包含私有数据的数据集时也可用。虽然仅在三分之一的数据集(CSS、PM和IM10)上观察到测试性能的一致改进,可能是因为这些问题容易被LLMs探索,但这种方法仅在1/77个测试用例中降低了模型性能。此外,这项工作将M6GP特征工程算法引入符号回归,表明它可以改善随机森林回归器的结果,并与其前身M3GP产生竞争性结果。

英文摘要:

Feature engineering is mandatory in the machine learning pipeline to obtain robust models. While evolutionary computation is well-known for its great results both in feature selection and feature construction, its methods are computationally expensive due to the large number of evaluations required to induce the final model. Part of the reason why these algorithms require a large number of evaluations is their lack of domain-specific knowledge, resulting in a lot of random guessing during evolution. In this work, we propose using Large Language Models (LLMs) as an initial feature construction step to add knowledge to the dataset. By doing so, our results show that the evolution can converge faster, saving us computational resources. The proposed approach only provides the names of the features in the dataset and the target objective to the LLM, making it usable even when working with datasets containing private data. While consistent improvements to test performance were only observed for one-third of the datasets (CSS, PM, and IM10), possibly due to problems being easily explored by LLMs, this approach only decreased the model performance in 1/77 test cases. Additionally, this work introduces the M6GP feature engineering algorithm to symbolic regression, showing it can improve the results of the random forest regressor and produce competitive results with its predecessor, M3GP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑