使用基于术语的表示衡量排序列表中的偏差
Measuring Bias in a Ranked List using Term-based Representations
AI总结:
针对现有NFaiRR指标衡量排序偏差的局限,提出基于术语表征的TExFAIR公平性指标,扩展AWRF框架,实验显示其与NFaiRR无强相关,衡量不同维度的公平性。
摘要:
在大多数近期研究中,文档排序中的性别偏差采用NFaiRR指标进行评估,该指标通过对每个排序文档的无偏差得分进行聚合来衡量排序列表中的偏差。这种衡量排序列表偏差的视角存在一个关键局限:排序列表中的单个文档可能存在偏差,但排序列表整体却能平衡各群体的表征。为解决这一问题,我们提出了一种名为TExFAIR(基于术语曝光的公平性)的新型指标,该指标基于通用公平性评估框架注意力加权排序公平性(AWRF)的两项全新扩展。TExFAIR基于排序列表中群体的基于术语的表征来评估公平性:(i)基于概率层面的术语级关联将文档关联到群体的明确定义,以及(ii)用于将非代表性文档计入排序列表公平性衡量的排序偏差折扣因子(RBDF)。我们在段落排序中的性别偏差衡量任务上对TExFAIR进行了评估,并研究了TExFAIR与NFaiRR之间的关系。我们的实验表明,TExFAIR与NFaiRR之间不存在强相关性,这表明TExFAIR衡量的是与NFaiRR不同的公平性维度。借助TExFAIR,我们对AWRF框架进行了扩展,使其能够在排序列表中的文档具备群体的基于术语的表征的场景下评估公平性。
英文摘要:
In most recent studies, gender bias in document ranking is evaluated with the NFaiRR metric, which measures bias in a ranked list based on an aggregation over the unbiasedness scores of each ranked document. This perspective in measuring the bias of a ranked list has a key limitation: individual documents of a ranked list might be biased while the ranked list as a whole balances the groups' representations. To address this issue, we propose a novel metric called TExFAIR (term exposure-based fairness), which is based on two new extensions to a generic fairness evaluation framework, attention-weighted ranking fairness (AWRF). TExFAIR assesses fairness based on the term-based representation of groups in a ranked list: (i) an explicit definition of associating documents to groups based on probabilistic term-level associations, and (ii) a rank-biased discounting factor (RBDF) for counting non-representative documents towards the measurement of the fairness of a ranked list. We assess TExFAIR on the task of measuring gender bias in passage ranking, and study the relationship between TExFAIR and NFaiRR. Our experiments show that there is no strong correlation between TExFAIR and NFaiRR, which indicates that TExFAIR measures a different dimension of fairness than NFaiRR. With TExFAIR, we extend the AWRF framework to allow for the evaluation of fairness in settings with term-based representations of groups in documents in a ranked list.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。