arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.02189cs.LG
收藏

一个探索大数据的系统:用于开放健康数据异常值检测的迭代k-means探照灯

A system for exploring big data: an iterative k-means searchlight for outlier detection on open health data

Fairleigh Dickinson University(费尔利迪金森大学)

A. Ravishankar Rao, Daniel Clarke, Subrata Garai, Soumyabrata Dey

AI总结:

提出基于迭代k-means探照灯与子集扫描的系统,通过拆分-应用-合并范式探索开放健康数据,识别出成本超支和自杀诊断增加等异常趋势。

摘要:

对大型且不断演变的数据集进行交互式探索具有挑战性,因为底层变量之间的关系可能未被完全理解。数据中可能隐藏着值得进一步探索和分析的趋势与模式。我们提出一个系统,使用探照灯技术系统地探索变量的多种组合并识别异常值。将迭代k-means聚类算法应用于通过数据库文献中的拆分-应用-合并范式派生的特征。异常值被识别为单例或小簇。该算法以探照灯方式扫过数据集。包含异常值的维度使用子集扫描技术与其他维度成对组合,以进一步深入了解异常值。我们通过分析纽约州发布的开放医疗保健数据来说明该系统。我们应用迭代k-means探照灯,随后进行子集扫描。识别出数据中的几个异常趋势,包括特定医院的成本超支,以及自杀等诊断的增加。这些构成了文献中的新发现,对监管机构、政策制定者和相关公民具有潜在用途。

英文摘要:

The interactive exploration of large and evolving datasets is challenging as relationships between underlying variables may not be fully understood. There may be hidden trends and patterns in the data that are worthy of further exploration and analysis. We present a system that methodically explores multiple combinations of variables using a searchlight technique and identifies outliers. An iterative k-means clustering algorithm is applied to features derived through a split-apply-combine paradigm used in the database literature. Outliers are identified as singleton or small clusters. This algorithm is swept across the dataset in a searchlight manner. The dimensions that contain outliers are combined in pairs with other dimensions using a susbset scan technique to gain further insight into the outliers. We illustrate this system by anaylzing open health care data released by New York State. We apply our iterative k-means searchlight followed by subset scanning. Several anomalous trends in the data are identified, including cost overruns at specific hospitals, and increases in diagnoses such as suicides. These constitute novel findings in the literature, and are of potential use to regulatory agencies, policy makers and concerned citizens.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑