一个基于特征的框架，用于检测来自原位传感器的水质数据中的技术异常值
A feature-based framework for detecting technical outliers in water-quality data from in situ sensors
摘要：
由于现场传感器的水质数据中的技术错误而产生的异常值可能会降低数据质量，并直接影响从随后的数据分析中得出的推断。然而，考虑到传感器产生的数据量和速度，通过手动监测进行离群值检测是不可行的。在这里，我们提出了一个自动化的框架，提供早期检测异常值的水质数据从原位传感器引起的技术问题。该框架首先用于识别数据特征，区分离群的实例从典型的行为。然后应用统计变换，使离群实例在变换后的数据空间中突出。然后将无监督离群值评分技术应用于转换后的数据空间，并使用基于极值理论的方法来计算每个潜在离群值的阈值。使用从流入澳大利亚大堡礁泻湖的河流中的原位传感器获得的两个数据集，我们表明，所提出的框架成功地识别了涉及浊度，电导率和河流水位突然变化的异常值，包括突然的峰值，突然的孤立下降和水位变化，同时保持非常低的错误检测率。我们在开源R包oddwater中实现了这个框架。
英文摘要：
Outliers due to technical errors in water-quality data from in situ sensors can reduce data quality and have a direct impact on inference drawn from subsequent data analysis. However, outlier detection through manual monitoring is unfeasible given the volume and velocity of data the sensors produce. Here, we proposed an automated framework that provides early detection of outliers in water-quality data from in situ sensors caused by technical issues.The framework was used first to identify the data features that differentiate outlying instances from typical behaviours. Then statistical transformations were applied to make the outlying instances stand out in transformed data space. Unsupervised outlier scoring techniques were then applied to the transformed data space and an approach based on extreme value theory was used to calculate a threshold for each potential outlier. Using two data sets obtained from in situ sensors in rivers flowing into the Great Barrier Reef lagoon, Australia, we showed that the proposed framework successfully identified outliers involving abrupt changes in turbidity, conductivity and river level, including sudden spikes, sudden isolated drops and level shifts, while maintaining very low false detection rates. We implemented this framework in the open source R package oddwater.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
