arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.09043cs.LG
收藏

在操纵下,某些AI模型是否更难审计?

Under manipulations, are some AI models harder to audit?

Univ Rennes(雷恩大学);Inria(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);IRISA(雷恩信息与自动化研究所);PEReN;LAAS/CNRS(法国国家科学研究中心自动化与系统分析实验室)

Augustin Godinot, Gilles Tredan, Erwan Le Merrer, Camilla Penzo, Francois Taïani

AI总结:

本文研究现实场景中鲁棒审计的可行性,证明平台用可拟合任何数据的模型时审计策略难优于随机抽样,通过拉德马赫复杂度关联审计可操纵性与模型容量,实证验证大容量模型更难鲁棒审计,细化审计局限并提出相关问题。

摘要:

审计人员需要可靠的方法来评估网络平台的合规性,但由于几乎无法访问平台使用的算法、实现或训练数据,该问题比简单的指标估计更复杂。在最近的防操纵审计框架内,本文研究了现实场景中鲁棒审计的可行性,其中模型具有大的容量。首先,我们证明了一个限制性结果:如果网络平台使用的模型可以拟合任何数据,那么在估计人口统计均等性等属性时,任何审计策略(无论是否主动)都无法优于随机抽样。为了更好地理解最先进的审计技术可能保持竞争力的条件,我们随后使用拉德马赫复杂度将审计的可操纵性与目标模型的容量联系起来。我们在容量递增的流行模型上进行了实证验证,实验证实了实践中常用的大容量模型特别难以进行鲁棒审计。这些结果细化了审计问题的局限性,并提出了关于模型容量与平台操纵审计尝试能力之间联系的诱人问题。

英文摘要:

Auditors need robust methods to assess the compliance of web platforms with the law. However, since they hardly ever have access to the algorithm, implementation, or training data used by a platform, the problem is harder than a simple metric estimation. Within the recent framework of manipulation-proof auditing, we study in this paper the feasibility of robust audits in realistic settings, in which models exhibit large capacities. We first prove a constraining result: if a web platform uses models that may fit any data, no audit strategy -- whether active or not -- can outperform random sampling when estimating properties such as demographic parity. To better understand the conditions under which state-of-the-art auditing techniques may remain competitive, we then relate the manipulability of audits to the capacity of the targeted models, using the Rademacher complexity. We empirically validate these results on popular models of increasing capacities, thus confirming experimentally that large-capacity models, which are commonly used in practice, are particularly hard to audit robustly. These results refine the limits of the auditing problem, and open up enticing questions on the connection between model capacity and the ability of platforms to manipulate audit attempts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑