arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.18673cs.DBcs.LG
收藏

实践中的开源漂移检测工具:来自两个用例的洞察

Open-Source Drift Detection Tools in Action: Insights from Two Use Cases

Software AG(软件AG公司);Granlund Oy(格兰隆德公司)

Rieke Müller, Mohamed Abdelaal, Davor Stjelja

AI总结:

本文提出 D3Bench 微基准,基于两个智能建筑用例评估 Evidently AI、NannyML 和 Alibi-Detect 的漂移检测功能与非功能表现。

摘要:

数据漂移在机器学习(ML)模型的生命周期中构成关键挑战,会影响其性能与可靠性。为应对这一挑战,我们提出一项名为 D3Bench 的微基准研究,用于评估开源漂移检测工具的有效性。D3Bench 利用来自两个智能建筑用例的真实数据,考察 Evidently AI、NannyML 和 Alibi-Detect 的能力。我们优先评估这些工具在识别和分析数据漂移方面的功能适用性。此外,我们还考虑了一套全面的非功能性标准,例如与 ML 管道的可集成性、对多种数据类型的适应性、用户友好性、计算效率以及资源需求。研究结果表明,Evidently AI 在通用数据漂移检测方面表现突出,而 NannyML 则擅长确定漂移发生的准确时间,并评估其对预测准确性随之产生的影响。

英文摘要:

Data drifts pose a critical challenge in the lifecycle of machine learning (ML) models, affecting their performance and reliability. In response to this challenge, we present a microbenchmark study, called D3Bench, which evaluates the efficacy of open-source drift detection tools. D3Bench examines the capabilities of Evidently AI, NannyML, and Alibi-Detect, leveraging real-world data from two smart building use cases.We prioritize assessing the functional suitability of these tools to identify and analyze data drifts. Furthermore, we consider a comprehensive set of non-functional criteria, such as the integrability with ML pipelines, the adaptability to diverse data types, user-friendliness, computational efficiency, and resource demands. Our findings reveal that Evidently AI stands out for its general data drift detection, whereas NannyML excels at pinpointing the precise timing of shifts and evaluating their consequent effects on predictive accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑