arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.14935cs.DBcs.AIcs.CEcs.LG
收藏

使用 Desbordante 解决数据质量问题:演示

Solving Data Quality Problems with Desbordante: a Demo

Universe Data LLC(宇宙数据有限责任公司);Saint-Petersburg University(圣彼得堡大学)
George Chernishev, Michael Polyntsov, Anton Chizhov, Kirill Stupakov, Ilya Shchuckin, Alexander Smirnov, Maxim Strutovsky, Alexey Shlyonskikh, Mikhail Firsov, S… 展开作者

George Chernishev, Michael Polyntsov, Anton Chizhov, Kirill Stupakov, Ilya Shchuckin, Alexander Smirnov, Maxim Strutovsky, Alexey Shlyonskikh, Mikhail Firsov, Stepan Manannikov, Nikita Bobrov, Daniil Goncharov, Ilia Barutkin, Vladislav Shalnev, Kirill Muraviev, Anna Rakhmukova, Dmitriy Shcheka, Anton Chernikov, Mikhail Vyrodov, Yaroslav Kurbatov, Maxim Fofanov, Sergei Belokonnyi, Pavel Anosov, Arthur Saliou, Eduard Gaisin, Kirill Smirnov

AI总结:

本文演示了开源数据剖析工具 Desbordante,通过其高效的 C++ 核心、Python 集成及解释能力,解决拼写错误检测、数据去重和异常检测等数据质量问题。

摘要:

数据剖析是现代数据驱动行业中的一个重要过程。其关键组成部分之一是复杂统计的发现与验证,包括函数依赖、数据约束、关联规则等。然而,大多数专注于复杂统计的现有数据剖析系统并未与当代数据科学家使用的工具进行适当集成。这在工业界采用这些工具时造成了重大障碍。此外,现有系统在设计时并未考虑工业级工作负载。最后,它们不旨在提供描述性解释,即为什么未发现给定模式。这是一个重大问题,因为理解特定模式缺失的潜在原因对于基于数据做出明智决策至关重要。因此,这些模式实际上是悬空的:它们的应用范围相当有限,很少被更广泛的公众使用。同时,正如我们将在本次演示中展示的那样,复杂统计可以有效地用于解决许多经典的数据质量问题。Desbordante 是一个开源数据剖析器,旨在弥合这一差距。它的构建强调工业应用:高效、可扩展、抗崩溃,并提供解释。此外,它通过将各种昂贵的操作卸载到 C++ 核心而不仅仅是挖掘,提供了无缝的 Python 集成。在本次演示中,我们展示了允许最终用户解决不同数据质量问题的几个场景。即,我们展示了拼写错误检测、数据去重和数据异常检测场景。

英文摘要:

Data profiling is an essential process in modern data-driven industries. One of its critical components is the discovery and validation of complex statistics, including functional dependencies, data constraints, association rules, and others. However, most existing data profiling systems that focus on complex statistics do not provide proper integration with the tools used by contemporary data scientists. This creates a significant barrier to the adoption of these tools in the industry. Moreover, existing systems were not created with industrial-grade workloads in mind. Finally, they do not aim to provide descriptive explanations, i.e. why a given pattern is not found. It is a significant issue as it is essential to understand the underlying reasons for a specific pattern's absence to make informed decisions based on the data. Because of that, these patterns are effectively rest in thin air: their application scope is rather limited, they are rarely used by the broader public. At the same time, as we are going to demonstrate in this presentation, complex statistics can be efficiently used to solve many classic data quality problems. Desbordante is an open-source data profiler that aims to close this gap. It is built with emphasis on industrial application: it is efficient, scalable, resilient to crashes, and provides explanations. Furthermore, it provides seamless Python integration by offloading various costly operations to the C++ core, not only mining. In this demonstration, we show several scenarios that allow end users to solve different data quality problems. Namely, we showcase typo detection, data deduplication, and data anomaly detection scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑