Web搜索引擎中具有高速下行能力的可扩展动态并行WebCrawler体系结构
Architecture of A Scalable Dynamic Parallel WebCrawler with High Speed Downloadable Capability for a Web Search Engine
摘要：
今天的万维网（WWW）已经成为一个巨大的信息海洋，它的规模每天都在增长。下载这些庞大数据中的一小部分就像在巨大的海洋中航行，这确实是一项具有挑战性的任务。为了从WWW上下载大量的数据，使爬行过程并行化已成为绝对必要的。在本文中，我们提供了一个动态的并行网络爬虫，命名为“WEB-SAILOR”，它提出了一个可扩展的方法，客户端-服务器模型的基础上，以加快下载过程中的Web搜索引擎在分布式域集特定的环境。WEB-SAILOR消除了多个爬虫程序重叠下载文档的可能性，甚至不会在几个并行的“客户端”爬行过程中产生通信开销。
英文摘要：
Today World Wide Web (WWW) has become a huge ocean of information and it is growing in size everyday. Downloading even a fraction of this mammoth data is like sailing through a huge ocean and it is a challenging task indeed. In order to download a large portion of data from WWW, it has become absolutely essential to make the crawling process parallel. In this paper we offer the architecture of a dynamic parallel Web crawler, christened as "WEB-SAILOR," which presents a scalable approach based on Client-Server model to speed up the download process on behalf of a Web Search Engine in a distributed Domain-set specific environment. WEB-SAILOR removes the possibility of overlapping of downloaded documents by multiple crawlers without even incurring the cost of communication overhead among several parallel "client" crawling processes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
