arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.08185cs.CRcs.CLcs.CYcs.NI
收藏

Web审查探测列表的自动生成

Automatic Generation of Web Censorship Probe Lists

Carnegie Mellon University(卡内基梅隆大学);EPFL(洛桑联邦理工学院);University of British Columbia(不列颠哥伦比亚大学)

Jenny Tang, Leo Alvarez, Arjun Brar, Nguyen Phong Hoang, Nicolas Christin

AI总结:

针对手动/众包生成Web审查探测列表耗时易错、扩展性差的问题,提出通过URL内容分析、主题扩展结合搜索引擎生成候选URL的自动生成方法,发现千余个原列表未覆盖的疑似被封域名,可助力大规模审查测量自动化。

摘要:

域名探测列表——用于确定需要探测哪些URL以排查Web审查——在互联网审查测量研究中发挥着关键作用。实际上,域名探测列表的规模和准确性限制了可检测的被审查页面集合;不准确的列表会导致对审查格局的认知不完整或产生有偏差的结果。以往生成域名探测列表的工作大多是手动完成或众包的,这种方法耗时、易出错,且无法很好地适配不断变化的审查格局。\n在本文中,我们探索了用于Web审查测量的、兼具全面性与时效性的探测列表自动生成方法。我们从包含多语言页面的各类现有测试列表中,选取139957个唯一URL的初始集合来生成新的候选页面。通过分析这些URL的内容(即执行主题和关键词提取)、扩展这些主题并将其作为输入提供给搜索引擎,我们的方法生成了覆盖35147个域名的119255个新URL。随后,我们在四个月的时间跨度内,从全球11个不同位置的服务器尝试访问每个URL,以此测试新的候选页面,检查其连通性和潜在的审查迹象。我们的测量结果显示,该方法发现了1400多个不在原始数据集中的、我们怀疑被封锁的域名。简而言之,自动更新探测列表是可行的,并且有助于进一步实现大规模审查测量的自动化。

英文摘要:

Domain probe lists--used to determine which URLs to probe for Web censorship--play a critical role in Internet censorship measurement studies. Indeed, the size and accuracy of the domain probe list limits the set of censored pages that can be detected; inaccurate lists can lead to an incomplete view of the censorship landscape or biased results. Previous efforts to generate domain probe lists have been mostly manual or crowdsourced. This approach is time-consuming, prone to errors, and does not scale well to the ever-changing censorship landscape. In this paper, we explore methods for automatically generating probe lists that are both comprehensive and up-to-date for Web censorship measurement. We start from an initial set of 139,957 unique URLs from various existing test lists consisting of pages from a variety of languages to generate new candidate pages. By analyzing content from these URLs (i.e., performing topic and keyword extraction), expanding these topics, and using them as a feed to search engines, our method produces 119,255 new URLs across 35,147 domains. We then test the new candidate pages by attempting to access each URL from servers in eleven different global locations over a span of four months to check for their connectivity and potential signs of censorship. Our measurements reveal that our method discovered over 1,400 domains--not present in the original dataset--we suspect to be blocked. In short, automatically updating probe lists is possible, and can help further automate censorship measurements at scale.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑