大语言模型是否具有独特且一致的人格？TRAIT：基于心理测量学为大语言模型设计的人格测试集
Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics
AI总结：
本研究推出基于心理测量学构建的LLM人格测试基准TRAIT，验证其在四项关键指标上优于现有测试，并揭示LLM具有受训练数据影响的独特一致人格，且现有提示技术对部分特质的激发效果有限。
摘要：
大语言模型（LLM）的最新进展使其作为对话智能体被应用于各个领域。我们不禁想问：能否像对人类一样，对这些智能体施测人格测试来分析其行为？我们推出了TRAIT，这是一个包含8000道选择题的全新基准，旨在评估LLM的人格特质。TRAIT基于两份经心理测量学验证的小型人类问卷——大五人格量表（Big Five Inventory, BFI）和短版黑暗三联征量表（Short Dark Triad, SD-3）构建，并借助ATOMIC-10X知识图谱扩展至多种现实场景。在信度和效度方面，TRAIT的表现也优于现有的LLM人格测试，在内容效度、内部效度、拒答率和信度四项关键指标上均取得最高分。借助TRAIT，我们揭示了关于LLM人格的两个重要发现：1）LLM表现出独特且一致的人格，这种人格高度受其训练数据（例如对齐调优所用数据）的影响；2）当前的提示技术在激发某些特质（如高精神病态或低尽责性）方面效果有限，表明该方向有待进一步研究。
英文摘要：
Recent advancements in Large Language Models (LLMs) have led to their adaptation in various domains as conversational agents. We wonder: can personality tests be applied to these agents to analyze their behavior, similar to humans? We introduce TRAIT, a new benchmark consisting of 8K multi-choice questions designed to assess the personality of LLMs. TRAIT is built on two psychometrically validated small human questionnaires, Big Five Inventory (BFI) and Short Dark Triad (SD-3), enhanced with the ATOMIC-10X knowledge graph to a variety of real-world scenarios. TRAIT also outperforms existing personality tests for LLMs in terms of reliability and validity, achieving the highest scores across four key metrics: Content Validity, Internal Validity, Refusal Rate, and Reliability. Using TRAIT, we reveal two notable insights into personalities of LLMs: 1) LLMs exhibit distinct and consistent personality, which is highly influenced by their training data (e.g., data used for alignment tuning), and 2) current prompting techniques have limited effectiveness in eliciting certain traits, such as high psychopathy or low conscientiousness, suggesting the need for further research in this direction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
