风险科学在AI公平性评估中的应用：原则、挑战与最佳实践
Applications of Risk Science to AI Fairness Evaluation: Principles, Challenges, and Best Practices
浏览论文内容
中文总结 AI 辅助
该研究探讨AI评估是否遵循风险科学原则，分析22种公平性指标的不足，通过简历筛选案例展示风险科学的应用，提出AI风险报告卡，推动AI社会影响评估进步。
中文摘要 AI 辅助
旨在描述日益普及的技术（尤其是与人工智能或其他算法系统相关的技术）潜在社会影响（例如风险）的学术工作，可能会产生超出其所属科学共同体的影响，因为普通社会本身就是这类研究的主要对象。然而，当前AI评估学术研究的实践是否遵循风险科学确立的原则和最佳实践仍是一个悬而未决的问题，风险科学旨在系统生成与理解、评估、沟通、管理和治理风险相关的知识。在本研究中，我们通过对声称评估用于招聘和就业相关任务的技术系统的偏见或公平性的学术作品进行文献综述，深入探讨了这一问题。通过分析22种常见的公平性评估指标及使用这些指标的研究，我们发现大多数指标仅刻画了与偏见或公平性相关后果的严重程度，却未遵循最佳实践来刻画这些后果发生的不确定性或严重程度估计的不确定性。接下来，我们对AI介导的简历筛选任务的公平性评估进行了案例研究，展示了如何将风险科学的原则融入此类评估中。最后，我们提出了AI风险报告卡（AI Risk Report Card），该工具便于向能够基于预测风险采取行动的利益相关者报告和沟通风险评估结果。这些活动的成果表明，在风险科学与AI评估交叉领域开展进一步研究，可通过建立共享框架，在科学共同体内部及外部评估和讨论AI风险，从而推动AI社会影响评估的进步。
英文摘要
Scholarly work which aims to describe potential societal impacts (e.g., risks) of proliferating technology (especially related to artificial intelligence or other algorithmic systems) is likely to have an impact beyond the scientific communities it was written for, given that general society itself is a primary object of study. However, it is an open question whether the current practices of AI evaluation scholarship follow the principles and best practices established by risk science, which aims to systematically generate knowledge related to understanding, assessing, communicating, managing, and governing risk. In this work, we examine this in depth by conducting a literature review of scholarly works purporting to evaluate the bias or fairness of technological systems used for tasks related to hiring and employment. Through analysis of 22 common fairness evaluation metrics and studies using them, we find that most characterize the severity of bias- or fairness-related consequences but do not follow best practices to characterize the uncertainty around either the occurrence of these consequences or severity estimates. Next, we conduct a case study of fairness evaluation for an AI-mediated resume screening task and demonstrate how principles of risk science can be incorporated into such an evaluation. Finally, we propose the AI Risk Report Card, which facilitates the reporting and communication of risk assessment results to stakeholders in positions to act based on the predicted risks. The outcomes of these activities suggest that further research at the convergence of risk science and AI evaluation can lead to advancements in AI assessments of societal impact by enabling shared frameworks to evaluate and discuss AI risks both within and outside of the scientific community.
发表机构
- University of Washington（华盛顿大学）
机构由 AI 辅助整理，请以论文原文为准。
