Censorship of the Internet is widespread around the world. As access to the web becomes increasingly ubiquitous, filtering of this resource becomes more pervasive. Transparency about specific content that citizens are denied access to is atypical. To counter this, numerous techniques for maintaining URL filter lists have been proposed by various individuals and organisations that aim to empirical data on censorship for benefit of the public and wider censorship research community.
We present a new approach for discovering filtered domains in different countries. This method is fully automated and requires no human interaction. The system uses web crawling techniques to traverse between filtered sites and implements a robust method for determining if a domain is filtered. We demonstrate the effectiveness of the approach by running experiments to search for filtered content in four different censorship regimes. Our results show that we perform better than the current state of the art and have built domain filter lists an order of magnitude larger than the most widely available public lists as of Jan 2018. Further, we build a dataset mapping the interlinking nature of blocked content between domains and exhibit the tightly networked nature of censored web resources.
Automatically Generating a Large, Culture-Specific Blocklist for China
互联网审查措施依赖于待测试网站的列表,或由第三方策划的“阻止列表”。不幸的是,这些列表中的许多都不是公开的,而且那些列表往往集中在一小部分主题上,而其他类型的网站和服务则未经测试。为了增加现有屏蔽列表中的网站并使其多样化,我们使用自然语言处理和搜索引擎自动发现在中国受到审查的更广泛的网站。使用这些技术,我们创建了一个Alexa Top 1,000之外的1125个网站的列表,涵盖中国政治,少数民族人权组织,受压迫的宗教等。重要的是,$\textit{none of the sites we discover are present on the current largest block list}$。我们开发的这份名单不仅大大扩展了目前互联网测量工具可以测试的网站,而且还加深了我们对中国审查内容性质的理解。我们已经发布了这个新的阻止列表和生成它的代码。
FilteredWeb: A Framework for the Automated Search-Based Discovery of Blocked URLs
已经提出了各种方法来创建和维护可能被过滤的URL的列表,以允许测量世界各地正在进行的互联网审查。虽然测试已知资源的过滤证据可能相对简单,但如果给定适当的有利位置,发现先前未知的过滤Web资源仍然是一个开放的挑战。我们提出了一个新的框架,通过使用自适应查询知名的搜索引擎自动发现过滤资源的过程。我们的系统应用信息检索算法来隔离已知过滤网页中的特征语言模式,然后将其用作网络搜索查询的基础。然后,检查这些查询的结果是否存在过滤证据,并将新发现的过滤资源反馈到系统中,以检测进一步过滤的内容。我们实施这个框架,适用于中国作为一个案例研究,表明这种方法在检测大量以前未知的过滤网页,使正在进行的检测互联网过滤,因为它的发展作出了重大贡献是显而易见的有效。截至2017年2月,我们的工具已被部署并用于发现中国境内1355个中毒域名-比最广泛使用的公共过滤器列表中包含的域名多30倍。其中,759个不在Alexa Top 1000域名列表中,这表明该框架能够找到更模糊的过滤内容。此外,我们对过滤URL的初步分析,以及用于发现它们的搜索词,进一步深入了解了目前在中国被屏蔽的内容的性质。