未抽样的真相：SLM 中的心理测量衡量的是提示伪影，而非心理构念
The Unsampled Truth: Quantifying Prompt Artifacts in LM Psychometrics
AI总结：
通过提示变异框架分离语义信号与提示伪影，发现小型语言模型在心理测量中主要反映提示遵从性而非模拟心理特质，并提供了诊断工具。
摘要：
当使用小型语言模型进行心理测量评估时，研究人员假设输出反映了语义推理。我们使用一个提示变异框架，在13个开放权重模型（0.6B到14B参数）上评估了这一前提，该框架将语义信号与提示伪影分离。通过系统地改变角色、指令、项目和选项符号，我们发现伪影方差经常压倒语义信号。在这些情况下，模型主要反映提示遵从性，而非模拟的心理特质。虽然这些发现限制了SLM在心理测量中的效用，但我们的框架提供了一个诊断工具，用于识别破坏性伪影并隔离语义理解，以用于未来的前沿模型研究。
英文摘要：
When prompting language models for psychometric assessment, researchers assume that the responses reflect the injected persona and the meaning of the survey item. We test this premise using a diagnostic design that crosses five semantically distinct baseline personas with five semantically equivalent variants of each of four prompt components (persona wording, task instruction, item wording, option symbol). Measuring the 1-Wasserstein distance between the resulting response distributions and partitioning the variation among the five components allows for the separation of target effects from prompt artifacts. We apply the framework to 13 open-weight small language models (0.6B to 14B) on the Big Five Inventory and the Short Dark Triad. We find that in most models, the task instruction and option symbol displace response distributions further than paraphrasing the persona description or the item itself. For a substantial share of items, the artifact share of explained variation exceeds 50%; non-semantic changes of the prompt account for more response variation than the baseline personas. Our framework lets researchers quantify these prompt artifacts before interpreting psychometric output.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
