JobFair：一个用于基准测试大语言模型中性别招聘偏见的框架
JobFair: A Framework for Benchmarking Gender Hiring Bias in Large Language Models
AI总结：
本文提出JobFair框架，基于劳动经济学等构建分层招聘偏见分类与严谨指标，分析10种先进LLMs的性别招聘偏见，发现多数存在对男性的反向偏见且多不随简历内容变化，还提供演示与数据集支持推广应用。
摘要：
大语言模型（LLMs）在招聘中的应用已推动相关立法行动以保护弱势人口群体。本文提出了一个用于基准测试大语言模型（LLMs）简历评分中分层性别招聘偏见的新型框架，揭示了反向性别招聘偏见和过度去偏见的显著问题。我们的贡献分为四点：首先，我们引入了一个基于劳动经济学、法律原则以及对当前偏见基准测试的批判的新构念：招聘偏见可分为两类——水平偏见（人口统计反事实群体间平均结果的差异）和离散偏见（人口统计反事实群体间结果方差的差异）；水平偏见可进一步细分为统计偏见（即随非人口统计内容变化）和基于偏好的偏见（即不受非人口统计内容影响而保持一致）。其次，该框架包含严谨的统计与计算型招聘偏见指标，例如评分后排名（RAS）、基于排名的影响比率、置换检验和固定效应模型。第三，我们分析了十种最先进的LLMs中的性别招聘偏见。十种LLMs中有七种在至少一个行业中表现出对男性的显著偏见。一项行业效应回归分析显示，医疗保健行业对男性的偏见最严重。此外，我们发现十种LLMs中有八种的偏见表现不随简历内容变化。这表明本文测量的偏见表现可能适用于其他具有不同简历质量的简历数据集。第四，我们提供了一个易用的演示程序和简历数据集，以支持该框架的采纳和实际应用，该框架可推广至其他社会特征和任务。
英文摘要：
The use of Large Language Models (LLMs) in hiring has led to legislative actions to protect vulnerable demographic groups. This paper presents a novel framework for benchmarking hierarchical gender hiring bias in Large Language Models (LLMs) for resume scoring, revealing significant issues of reverse gender hiring bias and overdebiasing. Our contributions are fourfold: Firstly, we introduce a new construct grounded in labour economics, legal principles, and critiques of current bias benchmarks: hiring bias can be categorized into two types: Level bias (difference in the average outcomes between demographic counterfactual groups) and Spread bias (difference in the variance of outcomes between demographic counterfactual groups); Level bias can be further subdivided into statistical bias (i.e. changing with non-demographic content) and taste-based bias (i.e. consistent regardless of non-demographic content). Secondly, the framework includes rigorous statistical and computational hiring bias metrics, such as Rank After Scoring (RAS), Rank-based Impact Ratio, Permutation Test, and Fixed Effects Model. Thirdly, we analyze gender hiring biases in ten state-of-the-art LLMs. Seven out of ten LLMs show significant biases against males in at least one industry. An industry-effect regression reveals that the healthcare industry is the most biased against males. Moreover, we found that the bias performance remains invariant with resume content for eight out of ten LLMs. This indicates that the bias performance measured in this paper might apply to other resume datasets with different resume qualities. Fourthly, we provide a user-friendly demo and resume dataset to support the adoption and practical use of the framework, which can be generalized to other social traits and tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
