CAPE：面向大型语言模型的上下文感知人格评估框架
CAPE: Context-Aware Personality Evaluation Framework for Large Language Models
AI总结：
本文提出 CAPE 框架，在人格心理测量中引入对话历史，并通过一致性指标评估 7 个 LLM，揭示上下文既提升一致性又导致人格偏移。
摘要：
传统上用于评估人类的心理测量测试，如今正被应用于大型语言模型（LLM），以评估其行为特征。然而，现有研究采用无上下文方法，孤立地回答每个问题以避免上下文影响。我们将其称为迪士尼世界测试，这是一种忽视真实世界应用的人工设定，而在真实应用中，对话历史会塑造回答。为弥合这一差距，我们提出了首个面向 LLM 的上下文感知人格评估（CAPE）框架，将先前的对话交互纳入其中。为全面分析上下文的影响，我们引入了新的指标，以量化 LLM 回答的一致性，而一致性是人类行为中的一项基本特征。我们在 7 个 LLM 上进行的详尽实验表明，对话历史通过上下文学习提升了回答一致性，但也会引发人格偏移，其中 GPT-3.5-Turbo 和 GPT-4-Turbo 表现出极端偏差。尽管 GPT 模型对问题顺序具有鲁棒性，Gemini-1.5-Flash 和 Llama-8B 却表现出显著敏感性。此外，GPT 模型的回答源于其内在人格特征以及先前交互，而 Gemini-1.5-Flash 和 Llama-8B 则严重依赖先前交互。最后，将我们的框架应用于角色扮演智能体（RPA）表明，依赖上下文的人格偏移提升了回答一致性，并更好地与人类判断保持一致。我们的代码和数据集公开发布于：https://github.com/jivnesh/CAPE
英文摘要：
Psychometric tests, traditionally used to assess humans, are now being applied to Large Language Models (LLMs) to evaluate their behavioral traits. However, existing studies follow a context-free approach, answering each question in isolation to avoid contextual influence. We term this the Disney World test, an artificial setting that ignores real-world applications, where conversational history shapes responses. To bridge this gap, we propose the first Context-Aware Personality Evaluation (CAPE) framework for LLMs, incorporating prior conversational interactions. To thoroughly analyze the influence of context, we introduce novel metrics to quantify the consistency of LLM responses, a fundamental trait in human behavior. Our exhaustive experiments on 7 LLMs reveal that conversational history enhances response consistency via in-context learning but also induces personality shifts, with GPT-3.5-Turbo and GPT-4-Turbo exhibiting extreme deviations. While GPT models are robust to question ordering, Gemini-1.5-Flash and Llama-8B display significant sensitivity. Moreover, GPT models response stem from their intrinsic personality traits as well as prior interactions, whereas Gemini-1.5-Flash and Llama--8B heavily depend on prior interactions. Finally, applying our framework to Role Playing Agents (RPAs) shows context-dependent personality shifts improve response consistency and better align with human judgments. Our code and datasets are publicly available at: https://github.com/jivnesh/CAPE
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
