arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.13347cs.CL
收藏

Craw4LLM:面向LLM预训练的高效网络爬取

Craw4LLM: Efficient Web Crawling for LLM Pretraining

Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所)

Shi Yu, Zhiyuan Liu, Chenyan Xiong

AI总结:

Craw4LLM提出基于LLM预训练偏好的网页优先级调度,仅爬取21%的URL即可达到同等下游性能,显著减少爬取浪费。

摘要:

网络爬取是大语言模型(LLMs)预训练数据的主要来源,但大多数爬取的网页因数据质量低而在预训练中被丢弃。本文提出Craw4LLM,一种基于LLM预训练偏好探索网络图的高效网络爬取方法。具体而言,它利用网页在LLM预训练中的影响力作为爬虫调度器的优先级分数,取代了基于图连通性的标准优先级。我们在包含来自商业搜索引擎索引的9亿个网页的网络图上进行的实验证明了Craw4LLM在获取高质量预训练数据方面的效率。仅爬取21%的URL,基于Craw4LLM数据预训练的LLM在下游任务上达到了与之前爬取相同的性能,显著减少了爬取浪费并减轻了网站负担。我们的代码公开于https://github.com/cxcscmu/Craw4LLM。

英文摘要:

Web crawl is a main source of large language models' (LLMs) pretraining data, but the majority of crawled web pages are discarded in pretraining due to low data quality. This paper presents Craw4LLM, an efficient web crawling method that explores the web graph based on the preference of LLM pretraining. Specifically, it leverages the influence of a webpage in LLM pretraining as the priority score of the web crawler's scheduler, replacing the standard graph connectivity based priority. Our experiments on a web graph containing 900 million webpages from a commercial search engine's index demonstrate the efficiency of Craw4LLM in obtaining high-quality pretraining data. With just 21% URLs crawled, LLMs pretrained on Craw4LLM data reach the same downstream performances of previous crawls, significantly reducing the crawling waste and alleviating the burdens on websites. Our code is publicly available at https://github.com/cxcscmu/Craw4LLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑