审计使用语言模型指导招聘决策的行为
Auditing the Use of Language Models to Guide Hiring Decisions
AI总结：
本文将检测人类判断偏见的对应实验用于审计多个最先进LLM的招聘评估，发现K-12教师申请中存在稳健的中等种族与性别差异，并讨论该审计方法的局限。
摘要：
随着大型语言模型（LLM）的快速发展，防范算法偏见的监管工作变得愈发紧迫；LLM是一类机器学习模型，在众多任务上能够取得与人类专家相匹敌的性能。这些举措的一个核心主题是算法“审计”，但当前法规以及科学文献几乎没有说明应如何开展此类评估。在此，我们提出并研究一种算法审计方法：对应实验，这是一种被广泛用于检测人类判断中偏见的工具。在就业情境中，对应实验通过实验性地操纵所提交申请材料中暗示申请者人口统计特征的元素（例如列出的姓名），来衡量种族和性别对决策的影响程度。我们应用该方法审计若干最先进LLM生成的候选人评估，使用了一个面向某大型公立学区K-12教学岗位申请的新颖语料库。我们发现存在中等程度种族和性别差异的证据；在改变输入模型的申请材料类型以及向LLM呈现任务的框架时，这一模式在很大程度上保持稳健。最后，我们讨论了对应实验用于审计算法时的一些重要局限。
英文摘要：
Regulatory efforts to protect against algorithmic bias have taken on increased urgency with rapid advances in large language models (LLMs), which are machine learning models that can achieve performance rivaling human experts on a wide array of tasks. A key theme of these initiatives is algorithmic "auditing," but current regulations -- as well as the scientific literature -- provide little guidance on how to conduct these assessments. Here we propose and investigate one approach for auditing algorithms: correspondence experiments, a widely applied tool for detecting bias in human judgements. In the employment context, correspondence experiments aim to measure the extent to which race and gender impact decisions by experimentally manipulating elements of submitted application materials that suggest an applicant's demographic traits, such as their listed name. We apply this method to audit candidate assessments produced by several state-of-the-art LLMs, using a novel corpus of applications to K-12 teaching positions in a large public school district. We find evidence of moderate race and gender disparities, a pattern largely robust to varying the types of application material input to the models, as well as the framing of the task to the LLMs. We conclude by discussing some important limitations of correspondence experiments for auditing algorithms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
