Blocks Architecture (BloArk)：面向维基百科修订历史的高效、高性价比增量式数据集架构
Blocks Architecture (BloArk): Efficient, Cost-Effective, and Incremental Dataset Architecture for Wikipedia Revision History
AI总结：
针对维基百科修订历史处理工具不足、资源消耗高的问题，提出高效数据处理架构BloArk，通过三层基础设施与构建、修改流水线实现格式转换与增量处理，支持下游NLP任务且已开源。
摘要：
维基百科（Wiki）是自然语言处理（NLP）应用中使用最广泛、公开可获取的资源之一。维基百科修订历史（WikiRevHist）记录了任意Wiki页面自首次修改以来的编辑顺序。尽管最新版本的维基百科已被广泛用作训练数据源，WikiRevHist同样可为NLP应用提供宝贵价值。但目前缺乏无需大量计算资源即可处理WikiRevHist的工具，使用者往往需要进行额外定制，还要花费额外时间适配他人的工作成果。为此，我们提出Blocks Architecture（BloArk，区块架构）——一种以效率为核心的数据处理架构，可减少处理WikiRevHist数据集时的运行时间、计算资源需求与重复工作。BloArk的基础设施由三部分组成：区块（blocks）、分段（segments）和仓库（warehouses）。在此之上，我们构建了核心数据处理流水线：构建器（builder）和修改器（modifier）。BloArk构建器将原始WikiRevHist数据集从XML格式转换为JSON Lines（JSONL）格式，以提升并发处理与存储效率。BloArk修改器可基于已构建的仓库执行增量修改，以提升现有数据库的利用率，降低复用他人工作成果的成本。最终，BloArk在处理维基百科修订历史以及对现有数据集进行增量修改以支持下游NLP应用场景两方面，均可轻松实现扩展。其源代码、文档与使用示例已在网上公开，以GPL-2.0许可证开源。
英文摘要：
Wikipedia (Wiki) is one of the most widely used and publicly available resources for natural language processing (NLP) applications. Wikipedia Revision History (WikiRevHist) shows the order in which edits were made to any Wiki page since its first modification. While the most up-to-date Wiki has been widely used as a training source, WikiRevHist can also be valuable resources for NLP applications. However, there are insufficient tools available to process WikiRevHist without having substantial computing resources, making additional customization, and spending extra time adapting others' works. Therefore, we report Blocks Architecture (BloArk), an efficiency-focused data processing architecture that reduces running time, computing resource requirements, and repeated works in processing WikiRevHist dataset. BloArk consists of three parts in its infrastructure: blocks, segments, and warehouses. On top of that, we build the core data processing pipeline: builder and modifier. The BloArk builder transforms the original WikiRevHist dataset from XML syntax into JSON Lines (JSONL) format for improving the concurrent and storage efficiency. The BloArk modifier takes previously-built warehouses to operate incremental modifications for improving the utilization of existing databases and reducing the cost of reusing others' works. In the end, BloArk can scale up easily in both processing Wikipedia Revision History and incrementally modifying existing dataset for downstream NLP use cases. The source code, documentations, and example usages are publicly available online and open-sourced under GPL-2.0 license.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
