arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2108.08905cs.LGstat.CO
收藏

统计学习实现领域无关数据质量评分

Statistical Learning to Operationalize a Domain Agnostic Data Quality Scoring

Indraprastha Institute of Information Technology(英德拉普拉斯塔信息技术学院)

Sezal Chug, Priya Kaushal, Ponnurangam Kumaraguru, Tavpritesh Sethi

摘要:

数据正在以难以想象的速度扩展,随着这种发展,数据质量的责任也随之而来。数据质量是指存在的信息的相关性,并有助于特定组织的决策和规划等各种操作。大多数情况下,数据质量是在特定的基础上测量的,因此,所开发的概念都没有提供任何实际应用。目前的实证研究是为了制定一个具体的自动化数据质量平台,以评估传入数据集的质量,并生成质量标签、评分和综合报告。我们利用来自healthdata.gov,opendata.nhs和人口统计和健康调查(DHS)计划的各种数据集来观察质量评分的变化,并使用主成分分析(PCA)制定标签。当前实证研究的结果揭示了一个包含9个质量成分的度量标准,即出处、数据集特征、均匀性、元数据耦合、缺失单元格和重复行的百分比、数据的偏度、分类列不一致的比率以及这些属性之间的相关性。该研究还提供了一个说明性的案例研究和验证的度量以下突变测试方法。这项研究提供了一个自动化平台,该平台采用传入的数据集和元数据来提供DQ评分、报告和标签。这项研究的结果将对数据科学家有用,因为这个质量标签的价值将在将数据部署到他/她各自的实际应用之前灌输信心。

英文摘要:

Data is expanding at an unimaginable rate, and with this development comes the responsibility of the quality of data. Data Quality refers to the relevance of the information present and helps in various operations like decision making and planning in a particular organization. Mostly data quality is measured on an ad-hoc basis, and hence none of the developed concepts provide any practical application. The current empirical study was undertaken to formulate a concrete automated data quality platform to assess the quality of incoming dataset and generate a quality label, score and comprehensive report. We utilize various datasets from healthdata.gov, opendata.nhs and Demographics and Health Surveys (DHS) Program to observe the variations in the quality score and formulate a label using Principal Component Analysis(PCA). The results of the current empirical study revealed a metric that encompasses nine quality ingredients, namely provenance, dataset characteristics, uniformity, metadata coupling, percentage of missing cells and duplicate rows, skewness of data, the ratio of inconsistencies of categorical columns, and correlation between these attributes. The study also provides an illustrative case study and validation of the metric following Mutation Testing approaches. This research study provides an automated platform which takes an incoming dataset and metadata to provide the DQ score, report and label. The results of this study would be useful to data scientists as the value of this quality label would instill confidence before deploying the data for his/her respective practical application.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑