利用神经网页爬取进行更清洁的预训练语料库整理
Cleaner Pretraining Corpus Curation with Neural Web Scraping
AI总结：
针对复杂网页导致传统爬虫难以提取干净文本的问题，本文提出NeuScraper神经网页爬虫，相较基线提升超过20%，可用于提升语言模型预训练语料质量。
摘要：
网络包含大规模、多样化且丰富的信息，可满足人类的信息检索需求。通过细致的数据收集、预处理和整理，网页能够作为语言模型预训练的基础数据资源。然而，面对不断革新且日益复杂的网页，基于规则或基于特征的网页爬虫正变得越来越力不从心。本文提出一种简单、快速且有效的神经网页爬虫（Neural web Scraper，NeuScraper），用于帮助从网页中提取主要且干净的文本内容。实验结果表明，NeuScraper相较基线爬虫实现了超过20%的提升，展现出其提取更高质量数据以促进语言模型预训练的潜力。所有代码均可在https://github.com/OpenMatch/NeuScraper获取。
英文摘要：
The web contains large-scale, diverse, and abundant information to satisfy the information-seeking needs of humans. Through meticulous data collection, preprocessing, and curation, webpages can be used as a fundamental data resource for language model pretraining. However, when confronted with the progressively revolutionized and intricate nature of webpages, rule-based/feature-based web scrapers are becoming increasingly inadequate. This paper presents a simple, fast, and effective Neural web Scraper (NeuScraper) to help extract primary and clean text contents from webpages. Experimental results show that NeuScraper surpasses the baseline scrapers by achieving more than a 20% improvement, demonstrating its potential in extracting higher-quality data to facilitate the language model pretraining. All of the code is available at https://github.com/OpenMatch/NeuScraper.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
