TextBenDS：一个通用的分布式文本数据基准
TextBenDS: a generic Textual data Benchmark for Distributed Systems
摘要：
使用加权方案提取top-k关键字和文档是文本挖掘和机器学习中用于不同分析和检索任务的流行技术。权重通常在数据预处理步骤中计算，因为更新和跟踪对数据集执行的所有修改的成本很高。此外，当仅分析数据集的子集时，会引入计算误差。因此，在大数据背景下，降低计算加权方案的运行时间，而不妨碍分析过程和机器学习算法的准确性至关重要。为了解决top-k关键字和文档的任务的这一需求，通常设计基准来比较分布式框架和数据库管理系统的各种配置内的加权方案。因此，我们提出了一个通用的面向文档的基准存储文本数据和构建加权方案（TextBendDS）。我们的基准测试提供了一个通用的数据模型，该模型采用多维方法来存储文本文档。我们还建议使用聚合查询与各种复杂性和选择性的建设长期加权方案，用于提取前k个关键字和文件。我们评估了Apache Hadoop生态系统中几个分布式环境下的查询的计算性能。我们的实验结果提供了有趣的见解。例如，MongoDB被证明具有最佳的整体性能，而Spark的执行时间几乎保持不变，无论权重方案如何。
英文摘要：
Extracting top-k keywords and documents using weighting schemes are popular techniques employed in text mining and machine learning for different analysis and retrieval tasks. The weights are usually computed in the data preprocessing step, as they are costly to update and keep track of all the modifications performed on the dataset. Furthermore, computation errors are introduced when analyzing only subsets of the dataset. Therefore, in a Big Data context, it is crucial to lower the runtime of computing weighting schemes, without hindering the analysis process and the accuracy of the machine learning algorithms. To address this requirement for the task of top-k keywords and documents, it is customary to design benchmarks that compare weighting schemes within various configurations of distributed frameworks and database management systems. Thus, we propose a generic document-oriented benchmark for storing textual data and constructing weighting schemes (TextBenDS). Our benchmark offers a generic data model designed with a multidimensional approach for storing text documents. We also propose using aggregation queries with various complexities and selectivities for constructing term weighting schemes, that are utilized in extracting top-k keywords and documents. We evaluate the computing performance of the queries on several distributed environments set within the Apache Hadoop ecosystem. Our experimental results provide interesting insights. As an example, MongoDB proves to have the best overall performance, while Spark's execution time remains almost the same, regardless of the weighting schemes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
