arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.02414cs.CYcs.LG
收藏

基于项目反应理论的公平性评估

Fairness Evaluation with Item Response Theory

RMIT University(RMIT大学);Data61, CSIRO(Data61、联邦科学与工业研究组织);Universität der Bundeswehr München(慕尼黑联邦国防军大学)

Ziqi Xu, Sevvandi Kandanaarachchi, Cheng Soon Ong, Eirini Ntoutsi

AI总结:

本文提出Fair-IRT框架,首次将项目反应理论应用于公平性评估,通过提取预测模型的公平预测能力及个体的区分度和难度参数,并利用项目特征曲线的平坦度解耦不公平性,实验和案例研究验证了其在分类与回归任务中的有效性。

摘要:

项目反应理论(Item Response Theory, IRT)已在教育心理测量学中被广泛用于评估学生能力,以及试题的难度和区分度。在此背景下,区分度特指试题有效区分不同能力水平学生的程度,并不包含任何与公平性相关的含义。近年来,IRT已被成功用于评估机器学习(Machine Learning, ML)模型的预测性能,但本文标志着其首次应用于公平性评估。在本文中,我们提出了一种新颖的Fair-IRT框架,用于评估一组预测模型在一组个体上的表现,同时提取特定参数,即做出公平预测的能力(预测模型的一个特征),以及影响预测结果的个体的区分度和难度。此外,我们进行了一系列实验,以全面理解这些参数对公平性评估的含义。针对特定个体,提供了项目特征曲线(Item Characteristic Curves, ICCs)的详细解释。我们提出使用ICCs的平坦度来解耦个体与预测模型之间的不公平性。实验证明了该框架作为公平性评估工具的有效性。两个真实世界案例研究展示了其在分类和回归任务中评估公平性的潜在应用。我们的论文与负责任网络(Responsible Web)方向高度契合,通过提出Fair-IRT框架来评估ML模型中的公平性,直接有助于发展一个更具包容性、公平性和可信赖的人工智能。

英文摘要:

Item Response Theory (IRT) has been widely used in educational psychometrics to assess student ability, as well as the difficulty and discrimination of test questions. In this context, discrimination specifically refers to how effectively a question distinguishes between students of different ability levels, and it does not carry any connotation related to fairness. In recent years, IRT has been successfully used to evaluate the predictive performance of Machine Learning (ML) models, but this paper marks its first application in fairness evaluation. In this paper, we propose a novel Fair-IRT framework to evaluate a set of predictive models on a set of individuals, while simultaneously eliciting specific parameters, namely, the ability to make fair predictions (a feature of predictive models), as well as the discrimination and difficulty of individuals that affect the prediction results. Furthermore, we conduct a series of experiments to comprehensively understand the implications of these parameters for fairness evaluation. Detailed explanations for item characteristic curves (ICCs) are provided for particular individuals. We propose the flatness of ICCs to disentangle the unfairness between individuals and predictive models. The experiments demonstrate the effectiveness of this framework as a fairness evaluation tool. Two real-world case studies illustrate its potential application in evaluating fairness in both classification and regression tasks. Our paper aligns well with the Responsible Web track by proposing a Fair-IRT framework to evaluate fairness in ML models, which directly contributes to the development of a more inclusive, equitable, and trustworthy AI.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑