使用Apache Spark的大规模文本处理管道
Large-scale text processing pipeline with Apache Spark
摘要：
在本文中，我们评估了Apache Spark的数据密集型机器学习问题。我们的用例重点关注美国各州立法机构随着时间的推移的政策扩散检测。由于计算强度的原因，以前关于政策扩散的工作一直无法在法案之间进行全对比较。作为替代，学者们研究了单一主题领域。我们提供了一个实现这个分析工作流作为一个分布式的文本处理管道与Spark框架和Scala应用程序编程接口。我们讨论了非结构化数据处理的挑战和策略，用于存储和有效访问的数据格式，以及大规模的图形处理。
英文摘要：
In this paper, we evaluate Apache Spark for a data-intensive machine learning problem. Our use case focuses on policy diffusion detection across the state legislatures in the United States over time. Previous work on policy diffusion has been unable to make an all-pairs comparison between bills due to computational intensity. As a substitute, scholars have studied single topic areas. We provide an implementation of this analysis workflow as a distributed text processing pipeline with Spark dataframes and Scala application programming interface. We discuss the challenges and strategies of unstructured data processing, data formats for storage and efficient access, and graph processing at scale.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
