发表机构
Saint-Petersburg University（圣彼得堡大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究旨在通过在Desbordante工具中支持概率函数依赖（pFD）来改进数据剖析。通过分析和实证研究pFD，评估其与近似函数依赖（AFD）差异，实现pFD发现算法并研究性能，与AFD算法比较，探索能否相互获取结果，为数据剖析提供新支持。
AI 中文摘要
数据剖析旨在从数据中提取复杂模式以用于进一步分析，并将数据用于数据清理、重复数据删除、异常检测等领域。函数依赖（FDs）是最著名的模式之一，但由于实际数据通常是脏数据，FDs的严格定义使其不适用于这些任务。因此有多种放宽FDs以支持脏数据的公式，近似函数依赖（AFD）最受欢迎，还有概率函数依赖（pFD）。本文旨在支持在C++实现的科学密集型、高性能开源数据剖析工具Desbordante中支持pFD。相比AFD，pFD研究较少。本文对pFD进行了分析和实证研究，评估不同pFD和AFD的差异，实现pFD发现算法并研究其运行时间和内存消耗，与AFD发现算法比较，还研究两种算法输出以了解能否用AFD发现算法获取pFD等。
英文摘要
Data profiling aims to extract complex patterns from data for further analysis and use that data in domains such as data cleaning, data deduplication, anomaly detection, and many more. Functional dependencies (FDs) are one of the most well-known patterns. However, they are poorly suited for these tasks, as real data is usually dirty, and the rigid definition of FDs does not allow algorithms to locate them. For this reason, there are several formulations aimed at relaxing FDs to support dirty data, with approximate functional dependency (AFD) being the most popular one. Another formulation is the Probabilistic Functional Dependency (pFD), which we aim to support inside Desbordante - a science-intensive, high-performance and open-source data profiling tool implemented in C++. However, pFDs are relatively poorly studied, compared to AFDs. In this paper we study pFDs, both analytically and empirically. We start by assessing how different pFDs and AFDs are by studying cases in which pFDs have an edge over AFDs. Then, we implement the algorithm for pFD discovery, as well as study its run time and memory consumption. We also compare it with an AFD discovery algorithm. Lastly, we study the output of both algorithms to learn whether or not it is possible to use AFD discovery algorithm to get pFDs and vice versa.
Journal ref35th Conference of Open Innovations Association (FRUCT), Tampere, Finland, 2024, pp. 158-169
DOI：10.23919/FRUCT61870.2024.10516409
