使用分层时间记忆和统计测试进行实时数据漂移和异常识别的混合框架
A Hybrid Framework for Real-Time Data Drift and Anomaly Identification Using Hierarchical Temporal Memory and Statistical Tests
AI总结：
本文提出结合HTM和SPRT的新型混合框架用于实时数据漂移检测与异常识别，并扩展了HTM在多维监督场景下的应用，实验证明其在准确性、适应性和计算效率上优于传统方法。
摘要：
数据漂移是指数据背后的生成模型随时间发生变化的现象。由于数据漂移，任何基于过去训练数据构建的模型都会随时间推移变得不再适用且不准确。因此，检测和控制数据漂移在机器学习模型中至关重要。分层时间记忆（Hierarchical Temporal Memory, HTM）是由Jeff Hawkins开发的机器学习模型，其灵感来源于人类大脑处理信息的方式。它是一种受生物学启发的记忆模型，在结构上类似于新皮层，其性能据称可与最先进的模型在检测时间序列数据异常时相媲美。HTM的另一个独特优势在于它不依赖于训练和测试周期；所有的学习都在流数据上在线进行，不需要单独的训练和测试周期。在序列学习范式中，序列概率比测试（Sequential Probability Ratio Test, SPRT）为在线学习和推理提供了一些独特的优势。本文提出了一种结合HTM和SPRT的新型混合框架，用于实时数据漂移检测和异常识别。与现有的数据漂移方法不同，我们的方法消除了频繁的重新训练，并确保了低误报率。目前HTM仅能处理一维或单变量数据。在第二项研究中，我们还提出了HTM在多维监督场景下用于异常检测的应用，即通过神经网络将多个HTM列（每个数据维度对应一个HTM列）的输出进行结合。实验评估表明，所提出的方法在准确性、适应性和计算效率方面优于传统的漂移检测技术，如柯尔莫哥洛夫-斯米尔诺夫（Kolmogorov-Smirnov, KS）检验、Wasserstein距离和群体稳定性指数（Population Stability Index, PSI）。我们的实验还为在电信等领域进行实时部署时优化超参数提供了见解。
英文摘要：
Data Drift is the phenomenon where the generating model behind the data changes over time. Due to data drift, any model built on the past training data becomes less relevant and inaccurate over time. Thus, detecting and controlling for data drift is critical in machine learning models. Hierarchical Temporal Memory (HTM) is a machine learning model developed by Jeff Hawkins, inspired by how the human brain processes information. It is a biologically inspired model of memory that is similar in structure to the neocortex, and whose performance is claimed to be comparable to state of the art models in detecting anomalies in time series data. Another unique benefit of HTMs is its independence from training and testing cycle; all the learning takes place online with streaming data and no separate training and testing cycle is required. In sequential learning paradigm, Sequential Probability Ratio Test (SPRT) offers some unique benefit for online learning and inference. This paper proposes a novel hybrid framework combining HTM and SPRT for real-time data drift detection and anomaly identification. Unlike existing data drift methods, our approach eliminates frequent retraining and ensures low false positive rates. HTMs currently work with one dimensional or univariate data. In a second study, we also propose an application of HTM in multidimensional supervised scenario for anomaly detection by combining the outputs of multiple HTM columns, one for each dimension of the data, through a neural network. Experimental evaluations demonstrate that the proposed method outperforms conventional drift detection techniques like the Kolmogorov-Smirnov (KS) test, Wasserstein distance, and Population Stability Index (PSI) in terms of accuracy, adaptability, and computational efficiency. Our experiments also provide insights into optimizing hyperparameters for real-time deployment in domains such as Telecom.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
