基于链接和内容的主题寻宝爬虫分析与评价
Analysis and Evaluation of the Link and Content Based Focused Treasure-Crawler
摘要：
随着Web的规模和分布呈指数级增长，为Web编制索引正成为搜索引擎的一项艰巨任务。目前，克服这个问题的最有效的已知方法是使用集中爬虫。聚焦爬虫应用适当的算法来检测Web上与其感兴趣的主题相关的页面。为此，我们提出了一个自定义的方法，使用特定的HTML元素的页面来预测所有的页面，有一个未访问的链接在当前页面的主题焦点。这些识别出的主题页面必须在以后根据它们与爬虫的主主题的相关性进行排序，以便进一步实际下载。在Treasure-Crawler中，我们使用了一个称为T-Graph的层次结构，它是一个示例性的指南，可以为每个未访问的链接分配适当的优先级得分。稍后将根据此优先级下载这些URL。本文概述了Treasure-Crawler系统的体系结构设计，并给出了该系统的实现、测试结果和性能评估。Treasure-Crawler是根据信息检索标准进行评估的，比如查全率和查准率，两者的值都接近0.5。取得这样的成果证明了所提出的方法的重要性。
英文摘要：
Indexing the Web is becoming a laborious task for search engines as the Web exponentially grows in size and distribution. Presently, the most effective known approach to overcome this problem is the use of focused crawlers. A focused crawler applies a proper algorithm in order to detect the pages on the Web that relate to its topic of interest. For this purpose we proposed a custom method that uses specific HTML elements of a page to predict the topical focus of all the pages that have an unvisited link within the current page. These recognized on-topic pages have to be sorted later based on their relevance to the main topic of the crawler for further actual downloads. In the Treasure-Crawler, we use a hierarchical structure called the T-Graph which is an exemplary guide to assign appropriate priority score to each unvisited link. These URLs will later be downloaded based on this priority. This paper outlines the architectural design and embodies the implementation, test results and performance evaluation of the Treasure-Crawler system. The Treasure-Crawler is evaluated in terms of information retrieval criteria such as recall and precision, both with values close to 0.5. Gaining such outcome asserts the significance of the proposed approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
