arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.07792cs.CLcs.AIcs.CY
收藏

大语言模型中社会偏见的双指标评估：来自少数族裔尼泊尔文化背景的证据

Dual-Metric Evaluation of Social Bias in Large Language Models: Evidence from an Underrepresented Nepali Cultural Context

Center for Artificial Intelligence Research Nepal（尼泊尔人工智能研究中心）

Ashish Pandey, Tek Raj Chhetri

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究通过双指标评估框架，分析了大语言模型在尼泊尔文化背景下对性别、种族和社会文化刻板印象的偏见，揭示了显性一致性偏见与隐性生成偏见的关系及影响因素。

摘要：

大语言模型（LLMs）日益影响全球数字生态系统，但其在未被代表的背景下持续社会和文化偏见的潜力仍不明确。本研究对七种最先进的LLMs进行了系统分析：GPT-4o-mini、Claude-3-Sonnet、Claude-4-Sonnet、Gemini-2.0-Flash、Gemini-2.0-Lite、Llama-3-70B和Mistral-Nemo在尼泊尔文化背景下。使用包含2400多个刻板印象和反刻板印象句子对的Croissant合规数据集，性别角色在社会领域中的表现，我们实施了评估框架，双指标偏见评估（DMBA），结合两个指标：（1）与偏见陈述的一致性，以及（2）刻板印象完成倾向。结果显示，模型表现出可测量的显性一致性偏见，平均偏见一致性范围从0.36到0.43，不同解码配置下，隐性完成偏率在0.740-0.755之间。重要的是，隐性完成偏见与温度呈非线性、U形关系，峰值出现在中等随机性（T=0.3）时，随着温度升高略有下降。不同解码设置下的相关性分析显示，显性一致性强烈与刻板印象句子一致性一致，但对隐性完成偏见的预测是弱的，通常是负的，表明生成偏见不被一致性指标所捕捉。敏感性分析显示，增加top-p会放大显性偏见，而隐性生成偏见保持基本稳定。领域层面的分析显示，隐性偏见在种族和社会文化刻板印象方面最强，而显性一致性偏见在性别和社会文化类别中相似，其中种族显示最低的显性一致性。这些发现突显了在未被代表的社会中，为LLMs开发文化扎根的数据集和去偏策略的必要性。

英文摘要：

Large language models (LLMs) increasingly influence global digital ecosystems, yet their potential to perpetuate social and cultural biases remains poorly understood in underrepresented contexts. This study presents a systematic analysis of representational biases in seven state-of-the-art LLMs: GPT-4o-mini, Claude-3-Sonnet, Claude-4-Sonnet, Gemini-2.0-Flash, Gemini-2.0-Lite, Llama-3-70B, and Mistral-Nemo in the Nepali cultural context. Using Croissant-compliant dataset of 2400+ stereotypical and anti-stereotypical sentence pairs on gender roles across social domains, we implement an evaluation framework, Dual-Metric Bias Assessment (DMBA), combining two metrics: (1) agreement with biased statements and (2) stereotypical completion tendencies. Results show models exhibit measurable explicit agreement bias, with mean bias agreement ranging from 0.36 to 0.43 across decoding configurations, and an implicit completion bias rate of 0.740-0.755. Importantly, implicit completion bias follows a non-linear, U-shaped relationship with temperature, peaking at moderate stochasticity (T=0.3) and declining slightly at higher temperatures. Correlation analysis under different decoding settings revealed that explicit agreement strongly aligns with stereotypical sentence agreement but is a weak and often negative predictor of implicit completion bias, indicating generative bias is poorly captured by agreement metrics. Sensitivity analysis shows increasing top-p amplifies explicit bias, while implicit generative bias remains largely stable. Domain-level analysis shows implicit bias is strongest for race and sociocultural stereotypes, while explicit agreement bias is similar across gender and sociocultural categories, with race showing the lowest explicit agreement. These findings highlight the need for culturally grounded datasets and debiasing strategies for LLMs in underrepresented societies.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑