arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.11366cs.DB
收藏

逐步数据清理建议以提高机器学习预测精度

Step-by-Step Data Cleaning Recommendations to Improve ML Prediction Accuracy

University of Amsterdam(阿姆斯特丹大学)

Sedir Mohammed, Felix Naumann, Hazar Harmouch

AI总结:

COMET通过逐步数据清理建议提升ML预测精度,优于多种现有方法。

摘要:

数据质量在机器学习(ML)应用中至关重要,因为数据中的错误会显著影响底层ML模型的预测精度。因此,数据清理是任何ML流水线中的重要组成部分。然而,在实际场景中,数据清理会带来显著成本,因为这通常涉及领域专家来配置和执行清理过程。因此,在数据清理过程中高效分配资源可以提高ML预测精度并控制开支。本文提出了COMET,一个旨在优化数据清理努力的系统。COMET为下一步清理哪个特征提供逐步建议,在资源受限的情况下最大化数据清理的效率。我们评估了COMET在各种数据集、ML算法和数据错误类型上的表现,证明了其鲁棒性和适应性。我们的结果表明,COMET在所有情况下均优于基于特征重要性、随机和另一种知名清理方法,其在ML预测精度上比所提出的基础线高出最高52个百分点,平均高出5个百分点。

英文摘要:

Data quality is crucial in machine learning (ML) applications, as errors in the data can significantly impact the prediction accuracy of the underlying ML model. Therefore, data cleaning is an integral component of any ML pipeline. However, in practical scenarios, data cleaning incurs significant costs, as it often involves domain experts for configuring and executing the cleaning process. Thus, efficient resource allocation during data cleaning can enhance ML prediction accuracy while controlling expenses. This paper presents COMET, a system designed to optimize data cleaning efforts for ML tasks. COMET gives step-by-step recommendations on which feature to clean next, maximizing the efficiency of data cleaning under resource constraints. We evaluated COMET across various datasets, ML algorithms, and data error types, demonstrating its robustness and adaptability. Our results show that COMET consistently outperforms feature importance-based, random, and another well-known cleaning method, achieving up to 52 and on average 5 percentage points higher ML prediction accuracy than the proposed baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑