In July 2023, New York City (NYC) initiated the first algorithm auditing system for commercial machine-learning systems. Local Law 144 (LL 144) mandates NYC-based employers using automated employment decision-making tools (AEDTs) in hiring to undergo annual bias audits conducted by an independent auditor. This paper examines lessons from LL 144 for other national algorithm auditing attempts. Through qualitative interviews with 16 experts and practitioners within the regime, we find that LL 144 has not effectively established an auditing regime. The law fails to clearly define key aspects, such as AEDTs and independent auditors, leading auditors, AEDT vendors, and companies using AEDTs to define the law's practical implementation in ways that failed to protect job applicants. Contributing factors include the law's flawed transparency-driven theory of change, industry lobbying narrowing the definition of AEDTs, practical and cultural challenges faced by auditors in accessing data, and wide disagreement over what constitutes a legitimate auditor, resulting in four distinct 'auditor roles.' We conclude with four recommendations for policymakers seeking to create similar bias auditing regimes, emphasizing clearer definitions, metrics, and increased accountability. By exploring LL 144 through the lens of auditors, our paper advances the evidence base around audit as an accountability mechanism, providing guidance for policymakers seeking to create similar regimes.
Learning About Algorithm Auditing in Five Steps: Scaffolding How High School Youth Can Systematically and Critically Evaluate Machine Learning Applications
University of Pennsylvania（宾夕法尼亚大学）；Teachers College, Columbia University（哥伦比亚大学教师学院）
Oxford Internet Institute, University of Oxford（牛津大学牛津互联网研究院）；Digital Ethics Center, Yale University（耶鲁大学数字伦理中心）；Harvard Law School（哈佛大学法学院）；Center for Information Technology Policy, Princeton University（普林斯顿大学信息技术政策中心）
Technion - Israel Institute of Technology（以色列理工学院）；ALTER (Association for Long Term Existence and Resilience)（ALTER（长期存在与韧性协会））；Transformative Futures Institute（变革性未来研究所）；National Intelligence University（国家情报大学）
Compliant But Unsatisfactory: The Gap Between Auditing Standards and Practices for Probabilistic Genotyping Software
University of California, Berkeley（加州大学伯克利分校）；Wright State University（莱特州立大学）；Forensic Bioinformatic Services, Inc.（法医生物信息学服务公司）；ELLIS Institute（ELLIS研究所）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Auditing large language models: a three-layered approach
University of Oxford（牛津大学）；Princeton University（普林斯顿大学）；Centre for the Governance of AI（人工智能治理中心）；Goethe University Frankfurt（法兰克福歌德大学）；University of Bologna（博洛尼亚大学）
