大型语言模型用于心理语言学合理性预测试
Large Language Models for Psycholinguistic Plausibility Pretesting
AI总结：
本研究探讨语言模型能否替代人类进行心理语言学材料的合理性预测试，发现GPT-4在多种结构上与人类判断高度相关，但仅在粗粒度判断时适用，细粒度判断仍不足。
摘要：
在心理语言学中，受控材料的创建对于确保研究结果仅归因于预期的操纵而不受外部因素影响至关重要。为实现这一目标，心理语言学家通常会对语言材料进行预测试，其中一种常见的预测试是征求人类评估者对特定句子的合理性判断。在本研究中，我们探讨语言模型（LMs）是否可用于生成这些合理性判断。我们考察了多种语言结构下的广泛语言模型，并评估其合理性判断是否与人类判断相关。我们发现，GPT-4的合理性判断在我们所考察的各种结构上与人类判断高度相关，而其他语言模型在常用的句法结构上与人类判断相关性良好。随后，我们测试了这种相关性是否意味着语言模型可以替代人类进行预测试。我们发现，当需要粗粒度合理性判断时，这种方法效果良好；但当需要细粒度判断时，即使是GPT-4也无法提供令人满意的区分能力。
英文摘要：
In psycholinguistics, the creation of controlled materials is crucial to ensure that research outcomes are solely attributed to the intended manipulations and not influenced by extraneous factors. To achieve this, psycholinguists typically pretest linguistic materials, where a common pretest is to solicit plausibility judgments from human evaluators on specific sentences. In this work, we investigate whether Language Models (LMs) can be used to generate these plausibility judgements. We investigate a wide range of LMs across multiple linguistic structures and evaluate whether their plausibility judgements correlate with human judgements. We find that GPT-4 plausibility judgements highly correlate with human judgements across the structures we examine, whereas other LMs correlate well with humans on commonly used syntactic structures. We then test whether this correlation implies that LMs can be used instead of humans for pretesting. We find that when coarse-grained plausibility judgements are needed, this works well, but when fine-grained judgements are necessary, even GPT-4 does not provide satisfactory discriminative power.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
