高维社交媒体数据流的并行聚类
Parallel clustering of high-dimensional social media data streams
摘要：
我们引入云DIKW作为一个分析环境，支持科学发现，通过集成的并行批处理和流处理，并将其应用到一个代表性的领域应用：社交媒体数据流聚类。最近的工作表明，可以通过使用反映文本内容和社交网络信息的高维向量表示数据点来生成高质量的聚类。由于相似性计算的高成本，即使是单遍算法的顺序实现也无法跟上真实世界流的速度。本文介绍了我们的努力，以满足实时社会流聚类通过并行化的限制。我们关注两个系统层面的问题。像Apache Storm这样的大多数流处理引擎都以有向非循环图的形式组织分布式工作器，这使得动态同步并行工作器的状态变得困难。我们通过使用发布-订阅消息传递系统创建单独的同步通道来应对这一挑战。由于高维向量的稀疏性，随着新数据点被分配到聚类，质心的大小快速增长。传统的同步，直接广播集群质心变得过于昂贵，限制了并行算法的可扩展性。我们解决这个问题的通信集群，而不是整个质心向量的动态变化。我们的算法在云DIKW下可以实时处理Twitter 10%的数据流，具有96路并行度。通过对Cloud DIKW的自然改进，包括在Harp项目中开发的高级集体通信技术，我们将能够以1000路并行实时处理完整的Twitter流。我们使用强大的通用软件子系统将使许多其他需要集成流和批处理数据分析的应用程序成为可能。
英文摘要：
We introduce Cloud DIKW as an analysis environment supporting scientific discovery through integrated parallel batch and streaming processing, and apply it to one representative domain application: social media data stream clustering. Recent work demonstrated that high-quality clusters can be generated by representing the data points using high-dimensional vectors that reflect textual content and social network information. Due to the high cost of similarity computation, sequential implementations of even single-pass algorithms cannot keep up with the speed of real-world streams. This paper presents our efforts to meet the constraints of real-time social stream clustering through parallelization. We focus on two system-level issues. Most stream processing engines like Apache Storm organize distributed workers in the form of a directed acyclic graph, making it difficult to dynamically synchronize the state of parallel workers. We tackle this challenge by creating a separate synchronization channel using a pub-sub messaging system. Due to the sparsity of the high-dimensional vectors, the size of centroids grows quickly as new data points are assigned to the clusters. Traditional synchronization that directly broadcasts cluster centroids becomes too expensive and limits the scalability of the parallel algorithm. We address this problem by communicating only dynamic changes of the clusters rather than the whole centroid vectors. Our algorithm under Cloud DIKW can process the Twitter 10% data stream in real-time with 96-way parallelism. By natural improvements to Cloud DIKW, including advanced collective communication techniques developed in our Harp project, we will be able to process the full Twitter stream in real-time with 1000-way parallelism. Our use of powerful general software subsystems will enable many other applications that need integration of streaming and batch data analytics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
