arXiv 2308.125782023-08-24cs.CLcs.AI

思维与表述：关于衡量大语言模型社会偏见重判不一致性的研究

Mind vs. Mouth: On Measuring Re-judge Inconsistency of Social Bias in Large Language Models

Tianjin University（天津大学）；China Mobile Communication Group Tianjin Co., Ltd.（中国移动通信集团天津有限公司）

Yachao Zhao, Bo Wang, Dongming Zhao, Kun Huang, Yan Wang, Ruifang He, Yuexian Hou

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

该研究聚焦大语言模型的外显与内隐社会偏见，提出两阶段方法发现“重判不一致性”现象，经ChatGPT和GPT-4实验验证其稳定性，为借助心理学理论探究大模型认知机制提供新视角。

摘要：

近期研究表明，预训练大语言模型（LLMs）具备与人类观测到的认知结构相似的认知构造，这促使研究者们探究LLMs的认知层面特性。本文聚焦于心理学中一种独特的双层认知结构——外显与内隐社会偏见。研究提出，个体的外显社会偏见（即个体在陈述中有意识表达的偏见）可能与其内隐社会偏见（代表个体无意识的偏见）存在差异。我们提出了一种两阶段方法，并在LLMs中发现了一种平行现象，即社会偏见的“重判不一致性”。在第一阶段，LLM被要求自动补全陈述，过程中可能融入内隐社会偏见；而在后续阶段，同一个LLM会对自己生成的带有偏见的陈述进行重新判断，却得出与之矛盾的结论。我们认为，这种重判不一致性可类比于人类未察觉的内隐社会偏见与可察觉的外显社会偏见之间的不一致性。针对ChatGPT和GPT-4开展的、围绕心理学中常见性别偏见的实验研究，证实了重判不一致性具有高度稳定的特性。这一发现或许表明，随着LLMs能力的增强，会涌现出多样化的认知结构。因此，运用心理学理论能够为理解LLMs中外显与内隐结构表达的底层机制提供更深入的洞见。

英文摘要：

Recent researches indicate that Pre-trained Large Language Models (LLMs) possess cognitive constructs similar to those observed in humans, prompting researchers to investigate the cognitive aspects of LLMs. This paper focuses on explicit and implicit social bias, a distinctive two-level cognitive construct in psychology. It posits that individuals' explicit social bias, which is their conscious expression of bias in the statements, may differ from their implicit social bias, which represents their unconscious bias. We propose a two-stage approach and discover a parallel phenomenon in LLMs known as "re-judge inconsistency" in social bias. In the initial stage, the LLM is tasked with automatically completing statements, potentially incorporating implicit social bias. However, in the subsequent stage, the same LLM re-judges the biased statement generated by itself but contradicts it. We propose that this re-judge inconsistency can be similar to the inconsistency between human's unaware implicit social bias and their aware explicit social bias. Experimental investigations on ChatGPT and GPT-4 concerning common gender biases examined in psychology corroborate the highly stable nature of the re-judge inconsistency. This finding may suggest that diverse cognitive constructs emerge as LLMs' capabilities strengthen. Consequently, leveraging psychological theories can provide enhanced insights into the underlying mechanisms governing the expressions of explicit and implicit constructs in LLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。