arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.10440cs.DB
收藏

一种新型多线程网络爬虫模型

A novel multi-threaded web crawling model

Weijie. Jiang

AI总结:

针对大规模网络数据采集需求,提出一种多线程网络爬虫模型,通过数据分片分配线程、爬取与解析均采用多线程并发的设计,经测试其性能较单线程方法有显著优化。

摘要:

本文提出了一种适用于大规模网络数据采集的新型网络爬虫模型。该模型首先将网络数据划分为若干子数据,每个子数据对应一个线程任务。在每个线程任务中,网络爬虫任务被并发执行,爬取到的数据会存储在缓冲队列中,等待进一步解析。解析过程同样被划分为多个线程。通过构建该模型并持续开展爬虫测试发现,与单线程方法相比,该模型的性能得到了显著优化。

英文摘要:

This paper proposes a novel model for web crawling suitable for large-scale web data acquisition. This model first divides web data into several sub-data, with each sub-data corresponding to a thread task. In each thread task, web crawling tasks are concurrently executed, and the crawled data are stored in a buffer queue, awaiting further parsing. The parsing process is also divided into several threads. By establishing the model and continuously conducting crawler tests, it is found that this model is significantly optimized compared to single-threaded approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑