arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.06492cs.LG
收藏

一种新的灵活的训练-测试分割算法:一种用于选择留出法、K折交叉验证和留出迭代的方法

A New Flexible Train-Test Split Algorithm, an approach for choosing among the Hold-out, K-fold cross-validation, and Hold-out iteration

University of Turin(都灵大学);University of Social Welfare and Rehabilitation Science(社会福利与康复大学);Brown University(布朗大学);Macquarie University(麦考瑞大学)

Zahra Bami, Ali Behnampour, Aniruddha Bora, Hassan Doosti

AI总结:

本文提出了一种灵活的训练-测试分割算法,通过评估不同验证策略对多种机器学习算法的影响,揭示了最佳验证方法取决于算法、数据集和评估指标的交互作用。

摘要:

选择合适的数据划分策略是机器学习中的关键步骤,但验证方法往往基于默认或传统设置选择,而没有考虑其对泛化能力和现实性能的影响。常见的方法如留出验证或固定k值的K折交叉验证经常仅基于经验实践应用。为了解决这个问题,我们提出了一种灵活的Python框架,系统地考察了不同验证策略对七个广泛使用的机器学习算法的预测性能的影响,包括决策树、K近邻、朴素贝叶斯变体、逻辑回归、校准线性支持向量机和基于直方图的梯度提升。该框架在多种验证方案下评估这些方法,包括从10%到90%的留出分割、k从3到15的K折交叉验证、重复留出和嵌套交叉验证。该框架应用于三个大小各异的生物医学数据集,并使用ROC-AUC、准确率和Matthews相关系数来评估性能。结果表明,没有任何单一的验证策略在所有算法和数据集中都优于其他方法,这表明最佳的验证方法取决于算法、数据集特征和评估指标之间的相互作用。

英文摘要:

Choosing an appropriate strategy for partitioning data into training and evaluation sets is a critical step in machine learning, yet validation methods are often selected using default or conventional settings without considering their impact on generalizability and real-world performance. Common approaches such as hold-out validation or k-fold cross-validation with fixed k values are frequently applied based solely on empirical practice. To address this issue, we propose a flexible Python-based framework that systematically examines how different validation strategies affect predictive performance across seven widely used machine learning algorithms, including Decision Trees, K-Nearest Neighbors, Naive Bayes variants, Logistic Regression, calibrated linear Support Vector Machines, and histogram-based gradient boosting. The framework evaluates these methods under a wide range of validation schemes, including hold-out splits from 10% to 90%, k-fold cross-validation with k between 3 and 15, repeated hold-out, and nested cross-validation. The framework is applied to three biomedical datasets of varying size, and performance is assessed using ROC-AUC, accuracy, and the Matthews correlation coefficient. The results show that no single validation strategy consistently outperforms others across all algorithms and datasets, indicating that optimal validation depends on the interaction between the algorithm, dataset characteristics, and evaluation metric.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑