不确定性驱动量化大语言模型中的社会偏见变化
Investigating Social Bias Changes in Quantized Language Models
AI总结：
研究发现量化过程导致大语言模型中社会偏见的翻转，且不确定性与量化强度影响偏见变化，需进行后续评估以确保可靠性。
摘要：
训练后量化降低了大语言模型的计算成本，但从根本上改变了其社会偏见，这种改变无法被聚合指标所捕捉。我们提出了对50个量化模型的首次大规模研究，这些模型在PostTrainingBiasBench上进行评估，该基准涵盖了13个封闭式和开放式偏见数据集。我们识别出一种现象，称为量化诱导的隐藏偏见翻转，即在量化后，多达21%的响应在偏见和非偏见状态之间翻转，尽管聚合偏见分数没有变化。这些翻转强烈受到模型不确定性的影响，其中不确定性高的响应比自信的响应更有可能改变，幅度为3-11倍。量化强度放大了这种影响，4位量化模型表现出比8位量化模型多4-6倍的行为变化。关键的是，这些变化在不同人口群体之间产生不对称影响，其中某些群体的偏见可能增加18.6%，而其他群体的偏见则减少14.1%，导致聚合结果看似中性。更大的模型没有表现出一致的鲁棒性优势，且群体特定的转变在不同模型家族中变化不可预测。我们的发现表明，压缩从根本上改变了偏见模式，需要关键的量化后评估和干预以确保实践中的可靠性。
英文摘要：
Post-training quantization reduces the memory needed to run large language models but alters their social biases in ways that aggregate metrics fail to capture. We present the first large-scale study of 50 quantized models evaluated on PostTrainingBiasBench, a unified benchmark of 13 closed- and open-ended bias datasets. We identify a phenomenon we term quantization-induced bias flipping, in which quantization causes models to change responses from biased to unbiased and vice versa, up to 21% of the time, despite no change in aggregate bias scores. These flips are strongly associated with model uncertainty, where the responses with high uncertainty are 3-11x more likely to change than the confident ones. Quantization strength amplifies this effect, with 4-bit quantized models exhibiting 4-6x more behavioral changes than 8-bit quantized models. Critically, these changes create asymmetric impacts across demographic groups, where bias can worsen by up to 18.6% for some groups while improving by 14.1% for others, yielding misleadingly neutral aggregate outcomes. Larger models show no consistent robustness advantage, and group-specific shifts vary unpredictably across model families. Our findings demonstrate that compression fundamentally alters bias patterns, requiring crucial post-quantization evaluation and interventions to ensure reliability in practice.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
