偏见能走多远？从预训练数据到对齐的偏见追溯
How far can bias go? Tracing bias from pretraining data to alignment
AI总结：
研究探讨预训练数据中的性别职业偏见如何在LLM中表现，通过零样本提示和词元共现分析，发现训练数据中的偏见被放大，指令微调部分缓解了代表性偏见，但整体性别刻板印象仍存在。
摘要：
随着LLM在用户应用中日益普及，解决持续加剧社会不平等的偏见至关重要。尽管已有大量工作测量或减轻这些模型中的偏见，但较少研究探讨其起源。本研究考察预训练数据中的性别-职业偏见与LLM中表现之间的相关性，重点分析Dolma数据集和OLMo模型。通过零样本提示和词元共现分析，探索训练数据中的偏见如何影响模型输出。研究发现，预训练数据中的偏见在模型输出中被放大。研究还考察提示类型、超参数和指令微调对偏见表达的影响，发现指令微调部分缓解了代表性偏见，但整体仍维持刻板的性别关联，而超参数和提示变化对偏见表达影响较小。本研究追溯LLM开发流程中的偏见，并强调在预训练阶段缓解偏见的重要性。
英文摘要：
As LLMs are increasingly integrated into user-facing applications, addressing biases that perpetuate societal inequalities is crucial. While much work has gone into measuring or mitigating biases in these models, fewer studies have investigated their origins. Therefore, this study examines the correlation between gender-occupation bias in pre-training data and their manifestation in LLMs, focusing on the Dolma dataset and the OLMo model. Using zero-shot prompting and token co-occurrence analyses, we explore how biases in training data influence model outputs. Our findings reveal that biases present in pre-training data are amplified in model outputs. The study also examines the effects of prompt types, hyperparameters, and instruction-tuning on bias expression, finding instruction-tuning partially alleviating representational bias while still maintaining overall stereotypical gender associations, whereas hyperparameters and prompting variation have a lesser effect on bias expression. Our research traces bias throughout the LLM development pipeline and underscores the importance of mitigating bias at the pretraining stage.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
