FilteredWeb:一个基于搜索的被屏蔽URL自动发现框架
FilteredWeb: A Framework for the Automated Search-Based Discovery of Blocked URLs
摘要:
已经提出了各种方法来创建和维护可能被过滤的URL的列表,以允许测量世界各地正在进行的互联网审查。虽然测试已知资源的过滤证据可能相对简单,但如果给定适当的有利位置,发现先前未知的过滤Web资源仍然是一个开放的挑战。我们提出了一个新的框架,通过使用自适应查询知名的搜索引擎自动发现过滤资源的过程。我们的系统应用信息检索算法来隔离已知过滤网页中的特征语言模式,然后将其用作网络搜索查询的基础。然后,检查这些查询的结果是否存在过滤证据,并将新发现的过滤资源反馈到系统中,以检测进一步过滤的内容。我们实施这个框架,适用于中国作为一个案例研究,表明这种方法在检测大量以前未知的过滤网页,使正在进行的检测互联网过滤,因为它的发展作出了重大贡献是显而易见的有效。截至2017年2月,我们的工具已被部署并用于发现中国境内1355个中毒域名-比最广泛使用的公共过滤器列表中包含的域名多30倍。其中,759个不在Alexa Top 1000域名列表中,这表明该框架能够找到更模糊的过滤内容。此外,我们对过滤URL的初步分析,以及用于发现它们的搜索词,进一步深入了解了目前在中国被屏蔽的内容的性质。
英文摘要:
Various methods have been proposed for creating and maintaining lists of potentially filtered URLs to allow for measurement of ongoing internet censorship around the world. Whilst testing a known resource for evidence of filtering can be relatively simple, given appropriate vantage points, discovering previously unknown filtered web resources remains an open challenge. We present a new framework for automating the process of discovering filtered resources through the use of adaptive queries to well-known search engines. Our system applies information retrieval algorithms to isolate characteristic linguistic patterns in known filtered web pages; these are then used as the basis for web search queries. The results of these queries are then checked for evidence of filtering, and newly discovered filtered resources are fed back into the system to detect further filtered content. Our implementation of this framework, applied to China as a case study, shows that this approach is demonstrably effective at detecting significant numbers of previously unknown filtered web pages, making a significant contribution to the ongoing detection of internet filtering as it develops. Our tool is currently deployed and has been used to discover 1355 domains that are poisoned within China as of Feb 2017 - 30 times more than are contained in the most widely-used public filter list. Of these, 759 are outside of the Alexa Top 1000 domains list, demonstrating the capability of this framework to find more obscure filtered content. Further, our initial analysis of filtered URLs, and the search terms that were used to discover them, gives further insight into the nature of the content currently being blocked in China.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。