基于Apache Spark的大规模网络嵌入
Large-Scale Network Embedding in Apache Spark
AI总结：
针对大规模图网络嵌入单机器难处理、计算效率低的问题，提出基于Apache Spark的递归分图并行分布式算法，线性成本获节点嵌入，性能与速度均优于现有方法，已落地腾讯游戏推荐场景。
摘要：
网络嵌入已被广泛应用于社交推荐与网络分析场景，例如图结构下的推荐系统和异常检测。但现有大多数方法无法高效处理大规模图，原因有两点：其一，图上的计算通常成本高昂；其二，图的规模或向量中间结果可能过大，导致单台机器难以处理。本文提出了一种基于Apache Spark的高效分布式大规模图网络嵌入算法，该算法将图递归划分为多个小型子图，以捕捉节点的内部与外部结构信息，随后并行计算每个子图的网络嵌入。最终通过聚合所有子图的输出，以线性成本得到节点嵌入。我们通过多组实验证明，所提方法能在数小时内处理拥有数十亿条边的图，速度至少比现有最优方法快4倍。此外，该方法在链接预测和节点分类任务上的性能分别提升了4.25%和4.27%。最后，我们将该算法部署在腾讯的两款在线游戏中，用于好友推荐和物品推荐场景，其运行速度较竞品最高提升91.11%，对应评估指标最高提升12.80%。
英文摘要：
Network embedding has been widely used in social recommendation and network analysis, such as recommendation systems and anomaly detection with graphs. However, most of previous approaches cannot handle large graphs efficiently, due to that (i) computation on graphs is often costly and (ii) the size of graph or the intermediate results of vectors could be prohibitively large, rendering it difficult to be processed on a single machine. In this paper, we propose an efficient and effective distributed algorithm for network embedding on large graphs using Apache Spark, which recursively partitions a graph into several small-sized subgraphs to capture the internal and external structural information of nodes, and then computes the network embedding for each subgraph in parallel. Finally, by aggregating the outputs on all subgraphs, we obtain the embeddings of nodes in a linear cost. After that, we demonstrate in various experiments that our proposed approach is able to handle graphs with billions of edges within a few hours and is at least 4 times faster than the state-of-the-art approaches. Besides, it achieves up to $4.25\%$ and $4.27\%$ improvements on link prediction and node classification tasks respectively. In the end, we deploy the proposed algorithms in two online games of Tencent with the applications of friend recommendation and item recommendation, which improve the competitors by up to $91.11\%$ in running time and up to $12.80\%$ in the corresponding evaluation metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
