arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.18542cs.LG
收藏

学习 Skinny Trees 的端到端特征选择方法

End-to-end Feature Selection Approach for Learning Skinny Trees

MIT(麻省理工学院)

Shibal Ibrahim, Kayhan Behdin, Rahul Mazumder

AI总结:

提出Skinny Trees端到端树集成特征选择方法,通过分组$\ell_0$-正则化器同时训练可微树并控制特征数量,在15个数据集上实现高压缩率与加速推理,且性能优于现有工具包。

摘要:

我们提出了一种新的基于优化的方法,用于树集成的特征选择,这是统计学和机器学习中的一个重要问题。流行的树集成工具包(例如 Gradient Boosted Trees 和 Random Forests)支持基于特征重要性分数的训练后特征选择,虽然非常流行,但已知它们存在缺点。我们提出了 Skinny Trees:一个用于树集成特征选择的端到端工具包,我们在训练树集成的同时控制所选特征的数量。我们基于优化的方法学习可微树的集成,并使用分组 $\ell_0$-正则化器同时执行特征选择。我们使用一阶方法进行优化,并为我们的方法提供收敛保证。我们使用从密集到稀疏的正则化调度方案,可以产生更具表达能力和更稀疏的树集成。在 15 个合成和真实世界数据集上,Skinny Trees 可以实现 $1.5\\!\times\\! -~620~\\!\times\\!$ 的特征压缩率,与密集树相比推理速度提高多达 $10\times$,且性能没有任何损失。Skinny Trees 在特征选择方面优于许多现有工具包,例如在 25\\% 特征预算的 AUC 性能方面,Skinny Trees 比 LightGBM 高出 $10.2\\%$(高达 $37.7\\%$),比 Random Forests 高出 $3\\%$(高达 $12.5\\%$)。

英文摘要:

We propose a new optimization-based approach for feature selection in tree ensembles, an important problem in statistics and machine learning. Popular tree ensemble toolkits e.g., Gradient Boosted Trees and Random Forests support feature selection post-training based on feature importance scores, while very popular, they are known to have drawbacks. We propose Skinny Trees: an end-to-end toolkit for feature selection in tree ensembles where we train a tree ensemble while controlling the number of selected features. Our optimization-based approach learns an ensemble of differentiable trees, and simultaneously performs feature selection using a grouped $\ell_0$-regularizer. We use first-order methods for optimization and present convergence guarantees for our approach. We use a dense-to-sparse regularization scheduling scheme that can lead to more expressive and sparser tree ensembles. On 15 synthetic and real-world datasets, Skinny Trees can achieve $1.5\!\times\! -~620~\!\times\!$ feature compression rates, leading up to $10\times$ faster inference over dense trees, without any loss in performance. Skinny Trees lead to superior feature selection than many existing toolkits e.g., in terms of AUC performance for 25\% feature budget, Skinny Trees outperforms LightGBM by $10.2\%$ (up to $37.7\%$), and Random Forests by $3\%$ (up to $12.5\%$).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑