Auditors need robust methods to assess the compliance of web platforms with the law. However, since they hardly ever have access to the algorithm, implementation, or training data used by a platform, the problem is harder than a simple metric estimation. Within the recent framework of manipulation-proof auditing, we study in this paper the feasibility of robust audits in realistic settings, in which models exhibit large capacities. We first prove a constraining result: if a web platform uses models that may fit any data, no audit strategy -- whether active or not -- can outperform random sampling when estimating properties such as demographic parity. To better understand the conditions under which state-of-the-art auditing techniques may remain competitive, we then relate the manipulability of audits to the capacity of the targeted models, using the Rademacher complexity. We empirically validate these results on popular models of increasing capacities, thus confirming experimentally that large-capacity models, which are commonly used in practice, are particularly hard to audit robustly. These results refine the limits of the auditing problem, and open up enticing questions on the connection between model capacity and the ability of platforms to manipulate audit attempts.
Exposing the Illusion of Fairness: Auditing Vulnerabilities to Distributional Manipulation Attacks
Institut de Mathématiques de Toulouse (IMT)(图卢兹数学研究所);Institut national de recherche en sciences et technologies du numérique (INRIA)(国家数字科学与技术研究院);Artificial and Natural Intelligence Toulouse Institute 2 (ANITI 2)(图卢兹人工智能与自然智能研究所2);Institut de Recherche en Informatique de Toulouse (IRIT)(图卢兹计算机研究所);Independent Researcher(独立研究员);CNRS IRL CROSSING(国家科学研究中心IRL CROSSING)
Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation
University of California, Berkeley(加州大学伯克利分校);Stanford University(斯坦福大学);New York University(纽约大学);Swiss Federal Institute of Technology Lausanne (EPFL)(洛桑联邦理工学院)
Multimodal Anomaly Detection based on Deep Auto-Encoder for Object Slip Perception of Mobile Manipulation Robots
Artificial Intelligence Institute, Seoul National University(首尔国立大学人工智能研究所);Dept. of Computer Science and Engineering, Seoul National University(首尔国立大学计算机科学与工程系);Interdisciplinary Program in Neuroscience, Seoul National University(首尔国立大学神经科学跨学科项目);Surromind Inc.(Surromind公司)