arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.08758cs.LGcs.CL
收藏

预测公平性的系统评价

Systematic Evaluation of Predictive Fairness

The University of Melbourne(墨尔本大学);Amazon Alexa AI(亚马逊Alexa人工智能);MBZUAI(穆罕默德·本·扎耶德人工智能大学)

Xudong Han, Aili Shen, Trevor Cohn, Timothy Baldwin, Lea Frermann

摘要:

在有偏差的数据集上训练时减轻偏差是一个重要的开放问题。已经提出了几种技术,然而,考虑到非常狭窄的数据条件,典型的评估制度非常有限。例如,对目标阶层不平衡和陈规定型观念的影响研究不足。为了解决这一差距,我们研究了多个任务中各种去偏方法的性能,包括二进制分类(Twitter情绪),多类分类(职业预测)和回归(效价预测)。通过大量的实验,我们发现,数据条件对相对模型性能有很大的影响,并且当仅在标准数据集上进行评估时,无法得出关于方法有效性的一般结论,这是公平性研究中的当前实践。

英文摘要:

Mitigating bias in training on biased datasets is an important open problem. Several techniques have been proposed, however the typical evaluation regime is very limited, considering very narrow data conditions. For instance, the effect of target class imbalance and stereotyping is under-studied. To address this gap, we examine the performance of various debiasing methods across multiple tasks, spanning binary classification (Twitter sentiment), multi-class classification (profession prediction), and regression (valence prediction). Through extensive experimentation, we find that data conditions have a strong influence on relative model performance, and that general conclusions cannot be drawn about method efficacy when evaluating only on standard datasets, as is current practice in fairness research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑