arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.05080cs.CL
收藏

大语言模型心理测量响应的共享潜在结构

The Shared Latent Structure of LLM Psychometric Responses

IDEAS Research Institute(IDEAS研究所);University of Warsaw(华沙大学)

Hubert Plisiecki, Sabina Siudaj, Kacper Dudzic, Anna Sterna, Maciej Gorski, Karolina Drozdz, Marcin Moskalewicz

AI总结:

该研究分析50个LLM在45份心理问卷中的响应,发现存在解释47.1%方差的共享潜在结构,其部分反映体验性自我归因与社会期望响应,揭示LLM心理测量响应存在超出问卷设计初衷的共享倾向。

摘要:

人类心理测量问卷正越来越多地用于表征大语言模型(LLM),但目前尚不清楚它们的响应究竟反映了什么。为回答该问题,我们采用心理测量方法分析了50个LLM在涵盖多个心理领域的45份问卷中的响应。我们发现,一个主导主成分解释了标准化、响应衍生的问卷层面原始得分中47.1%的方差,揭示了不同工具间存在大量共享结构。我们通过语义分析、响应者框架和人工注释探究其含义:语义分析表明,该成分部分反映了体验性自我归因,即模型认可描述主观状态和身体体验的陈述的难易程度;一个强调在人类模拟下分歧减少的条目的模型得分与该成分强相关(r=0.864)。对289个匹配条目的人工评分显示,在相互调整并控制其他语义属性后,内在生活内容与条目-成分相关性呈正相关,社会期望与该相关性呈负相关。总体而言,六个评分属性解释了Fisher变换后的条目-成分相关性中22.1%的方差,支持涉及体验性自我归因和社会期望响应的部分解释。这些发现表明,LLM的心理测量轮廓具有共享的响应倾向,需要对其进行超出问卷设计初衷测量的人类特质的解读。

英文摘要:

Human psychometric questionnaires are increasingly used to characterize large language models (LLMs), but it remains unclear what exactly their responses reflect. To answer this question we analyze responses from 50 LLMs across 45 questionnaires spanning multiple psychological domains using psychometric methodology. We found a dominant principal component explaining 47.1% of variance across standardized, response-derived questionnaire-level primary scores, revealing substantial shared structure across instruments. We investigate its meaning through semantic analysis, respondent framing, and human annotations. Semantic analysis suggests that the component partly reflects experiential self-attribution: how readily models endorse statements describing subjective states and bodily experiences. A model score emphasizing items whose disagreement decreases under human simulation correlates strongly with the component ($r=0.864$). Human ratings of 289 matched items show that inner-life content is positively associated, and social desirability negatively associated, with item-component correlations after mutual adjustment and controls for other semantic properties. Together, six rated properties explain 22.1% of variance in Fisher-transformed item-component correlations, supporting a partial interpretation involving both experiential self-attribution and socially desirable responding. These findings suggest that LLM psychometric profiles share response tendencies that require interpretation beyond the human traits their questionnaires were designed to measure.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑