替代Web存档服务器加快数据到洞察周期的案例
The Case For Alternative Web Archival Formats To Expedite The Data-To-Insight Cycle
摘要：
WARC文件格式被广泛用于Web存档，以保存收集的Web内容供将来使用。随着网络档案的快速增长以及将这些档案重新用作统计和分析研究的大数据源的兴趣越来越大，将这些数据转化为见解的速度变得至关重要。在本文中，我们表明，WARC格式进行批处理工作负载的显着性能损失。我们跟踪这些惩罚的根本原因，其数据结构，编码和寻址方法。然后，我们进行对照实验来说明这些问题有多严重。实际上，只需将WARC文件重新格式化为Parquet或Avro格式，就可以获得一到两个数量级的性能增益。虽然这些结果并不一定构成对Avro或Parquet的认可，但Web存档社区考虑用更有效的Web存档格式取代WARC的时机已经到来。
英文摘要：
The WARC file format is widely used by web archives to preserve collected web content for future use. With the rapid growth of web archives and the increasing interest to reuse these archives as big data sources for statistical and analytical research, the speed to turn these data into insights becomes critical. In this paper we show that the WARC format carries significant performance penalties for batch processing workload. We trace the root cause of these penalties to its data structure, encoding, and addressing method. We then run controlled experiments to illustrate how severe these problems can be. Indeed, performance gain of one to two orders of magnitude can be achieved simply by reformatting WARC files into Parquet or Avro formats. While these results do not necessarily constitute an endorsement for Avro or Parquet, the time has come for the web archiving community to consider replacing WARC with more efficient web archival formats.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
