基于稠密次张量挖掘的万亿次尺度多方面数据群体异常检测
Detecting Group Anomalies in Tera-Scale Multi-Aspect Data via Dense-Subtensor Mining
摘要：
我们如何检测大规模多方面数据中的欺诈性锁步行为（即，张量）？当数据太大而无法放入内存甚至磁盘时，我们能检测到它吗？过去的研究表明，真实世界张量中的稠密子张量（例如，社交媒体、维基百科、TCP转储等）发出异常或欺诈行为的信号，例如转发提升、僵尸活动和网络攻击。因此，各种方法，包括张量分解和搜索，已被提出用于快速和准确地检测稠密子张量。然而，现有的方法具有低精度，或者它们假设张量足够小以适合主存，这在许多现实世界的应用中是不现实的，例如社交媒体和网络。为了克服这些限制，我们提出了D-CUBE，一种基于磁盘的稠密子张量检测方法，它也可以在多台机器上以分布式方式运行。与最先进的方法相比，D-CUBE具有（1）内存效率：需要的内存最多减少1，561倍，处理的数据最多增加1，000倍（2.6TB），（2）快速：由于其接近线性的可扩展性，速度快7倍，（3）可证明准确：保证检测到的子张量的密度，以及（4）有效：从TCP转储中发现网络攻击，并最准确地同步评级数据中的行为。
英文摘要：
How can we detect fraudulent lockstep behavior in large-scale multi-aspect data (i.e., tensors)? Can we detect it when data are too large to fit in memory or even on a disk? Past studies have shown that dense subtensors in real-world tensors (e.g., social media, Wikipedia, TCP dumps, etc.) signal anomalous or fraudulent behavior such as retweet boosting, bot activities, and network attacks. Thus, various approaches, including tensor decomposition and search, have been proposed for detecting dense subtensors rapidly and accurately. However, existing methods have low accuracy, or they assume that tensors are small enough to fit in main memory, which is unrealistic in many real-world applications such as social media and web. To overcome these limitations, we propose D-CUBE, a disk-based dense-subtensor detection method, which also can run in a distributed manner across multiple machines. Compared to state-of-the-art methods, D-CUBE is (1) Memory Efficient: requires up to 1,561X less memory and handles 1,000X larger data (2.6TB), (2) Fast: up to 7X faster due to its near-linear scalability, (3) Provably Accurate: gives a guarantee on the densities of the detected subtensors, and (4) Effective: spotted network attacks from TCP dumps and synchronized behavior in rating data most accurately.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
