arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.14583cs.AI
收藏

LLM偏差评估:职业与犯罪场景中的性别、种族和年龄差异

LLM Bias Evaluation: Gender, Racial, and Age Disparities in Occupational and Crime Scenarios

New York University(纽约大学);Northeastern University(东北大学);Washington University in St. Louis(圣路易斯华盛顿大学)

Vishal Mirza, Rahul Kulkarni, Aakanksha Jadhav

AI总结:

本文评估了2024年四大领先LLM在职业和犯罪场景中的性别、种族和年龄偏差,发现去偏努力常导致新的公平性权衡,即“去偏悖论”。

摘要:

LLM偏差评估至关重要,因为大型语言模型(LLM)越来越多地影响高风险决策。本文对领先LLM中的性别、种族和年龄差异进行了全面评估,揭示出去偏努力常常创造新的公平性权衡。近年来LLM的进展显著,但由于各种限制,企业广泛采用仍然有限。本文考察了LLM中的偏差——这是一个影响其可用性、可靠性和公平性的关键问题。我们的研究评估了2024年发布的四个领先LLM(Gemini 1.5 Pro、Llama 3 70B、Claude 3 Opus和GPT-4o)在职业场景中的性别偏差以及犯罪场景中的性别、年龄和种族偏差。结果显示,LLM在各种职业中描绘女性角色的频率往往高于男性,与美国劳工统计局数据相比偏差达37%。在犯罪场景中,与美国联邦调查局数据的偏差在性别上为54%,种族上为28%,年龄上为17%。关键的是,我们观察到减少性别和种族偏差的努力常常导致过度偏向某一子类的结果,可能加剧差异——这种“去偏悖论”凸显了当前偏差缓解技术的局限性,并强调了更有效方法的必要性。

英文摘要:

LLM bias evaluation is critical as large language models (LLMs) increasingly influence high-stakes decisions. This paper provides a comprehensive assessment of gender, racial, and age disparities in leading LLMs, revealing that debiasing efforts often create new fairness trade-offs. Recent advancements in LLMs have been notable, yet widespread enterprise adoption remains limited due to various constraints. This paper examines bias in LLMs - a crucial issue affecting their usability, reliability, and fairness. Our study evaluates gender bias in occupational scenarios and gender, age, and racial bias in crime scenarios across four leading LLMs released in 2024: Gemini 1.5 Pro, Llama 3 70B, Claude 3 Opus, and GPT-4o. Findings reveal that LLMs often depict female characters more frequently than male ones in various occupations, showing a 37% deviation from US BLS data. In crime scenarios, deviations from US FBI data are 54% for gender, 28% for race, and 17% for age. Critically, we observe that efforts to reduce gender and racial bias often lead to outcomes that may over-index one sub-class, potentially exacerbating disparities - a "debiasing paradox" that highlights the limitations of current bias mitigation techniques and underscores the need for more effective approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑