Infini-News:13亿篇Common Crawl新闻文章的高效可查询访问
Infini-News: Efficiently Queryable Access to 1.3 Billion Processed Common Crawl News Articles
AI总结:
本文提出Infini-News,通过构建全文索引和检索工具,为研究者提供了13亿篇Common Crawl新闻文章的高效访问方式,包括文本清洗、多语言检测、地理归属和高效检索功能,降低了纵向跨国媒体研究的门槛。
摘要:
大规模新闻语料库支持计算社会科学和自然语言处理中的广泛研究,但访问仍然受限:商业档案施加了昂贵的成本和许可限制,而开放替代方案如Common Crawl的CC-News需要TB级存储和计算密集型处理。我们提出了Infini-News,一个用于整个CC-News档案(2016年8月至最新可用快照)的检索工具包和索引。我们的贡献有三方面:首先,我们提取、清洗超过13.5亿篇文章的文本,并解析结构化元数据。其次,我们通过三种前沿语言分类器(GlotLID、lingua和CommonLingua)丰富语料库,并通过多源地理归属确定83.4%的文章的国家来源(涵盖222个国家)。第三,我们构建了Infini-gram索引:后缀数组结构,使研究者能够以亚秒级时间在全文档案中搜索任意文本模式。这些资源降低了纵向、跨国媒体研究的门槛。
英文摘要:
Large-scale news corpora support a wide range of research in Computational Social Science and NLP, yet access remains constrained: commercial archives impose prohibitive costs and licensing restrictions, while open alternatives like Common Crawl's CC-News require terabyte-scale storage and computationally intensive processing. We present Infini-News, a retrieval toolkit and index for the entire CC-News archive from August 2016 to the latest available snapshot. Our contributions are threefold. First, we extract, clean the text, and parse the structured metadata of over 1.35B articles. Second, we enrich the corpus with language detection using three frontier language classifiers (GlotLID, lingua, and CommonLingua), and with multi-source geographic attribution that resolves a country of origin for 83.4% of articles across 222 countries. Third, we construct Infini-gram indexes: suffix-array structures that let researchers search the full archive for arbitrary text patterns in sub-second time. Together, these resources lower the barrier to longitudinal, cross-national media research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。