GPT与偏见：理解大语言模型习得表征的稀疏方法
GPT and Prejudice: A Sparse Approach to Understanding Learned Representations in Large Language Models
AI总结：
针对大语言模型偏见溯源难的问题，提出结合LLM与稀疏自编码器的流水线，通过小说语料训练的GPT风格模型验证，该方法可追踪训练中主题编码过程，为审计模型表征中的文化假设提供可扩展框架。
摘要：
大语言模型（LLMs）在海量非结构化语料上训练，人们尚不清楚其吸收并复现了哪些社会模式与偏见。现有评估通常考察模型输出或激活值，却极少将其与预训练数据建立关联。本文提出一种将大语言模型与稀疏自编码器（SAEs）结合的流水线，用于追踪训练过程中不同主题的编码方式。作为对照案例研究，我们在10位女性作家创作的37部19世纪小说语料上训练了一个GPT风格模型，该语料围绕性别、婚姻、阶级、道德等主题展开。通过在各网络层应用SAEs，并使用11个社会与道德类别进行探测，我们将稀疏特征映射为人类可解释的概念。分析揭示了稳定的主题主干（最突出的是性别与亲属关系相关主题），并展示了关联如何随网络深度扩展并产生纠缠。更广泛地说，我们认为LLM+SAEs流水线为审计数据中的文化假设如何嵌入模型表征提供了可扩展框架。
英文摘要：
Large Language Models (LLMs) are trained on massive, unstructured corpora, making it unclear which social patterns and biases they absorb and later reproduce. Existing evaluations typically examine outputs or activations, but rarely connect them back to the pre-training data. We introduce a pipeline that couples LLMs with sparse autoencoders (SAEs) to trace how different themes are encoded during training. As a controlled case study, we trained a GPT-style model on 37 nineteenth-century novels by ten female authors, a corpus centered on themes such as gender, marriage, class, and morality. By applying SAEs across layers and probing with eleven social and moral categories, we mapped sparse features to human-interpretable concepts. The analysis revealed stable thematic backbones (most prominently around gender and kinship) and showed how associations expand and entangle with depth. More broadly, we argue that the LLM+SAEs pipeline offers a scalable framework for auditing how cultural assumptions from the data are embedded in model representations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
