利用外部评估识别大语言模型中的多重人格
Identifying Multiple Personalities in Large Language Models with External Evaluation
AI总结：
本文提出用外部评估方法替代传统自评测试，通过微调Llama2-7B作为MBTI预测器分析LLMs对情境问题的回应，发现LLMs在不同角色下表现出显著不同的人格，揭示了其与人类人格的根本差异。
摘要：
随着大语言模型（LLMs）迅速融入人类日常应用，围绕LLMs行为的诸多社会与伦理关切随之出现。理解LLMs行为的一种方式是分析其人格。许多近期研究使用为人类设计的自评测试来量化LLMs的人格，但许多批评质疑这些自评测试在应用于LLMs时的适用性和可靠性。在本文中，我们采用一种替代性人格测量方法——我们称之为外部评估方法——来研究LLMs的人格，该方法不是用李克特量表形式的多项选择题来提示LLMs，而是通过使用外部机器学习模型分析LLMs对开放式情境问题的回应来评估其人格。我们首先微调了一个Llama2-7B模型作为MBTI人格预测器，该模型在分析LLMs回应方面优于当前最先进的模型。然后，我们用情境问题提示LLMs，要求它们分别生成Twitter帖子和评论，以评估它们在扮演两种不同角色时的人格。利用外部人格评估方法，我们发现LLMs在生成帖子与生成评论时所获得的人格类型存在显著差异，而人类在这两种不同情境中表现出稳定一致的人格特征。这表明LLMs可以根据不同场景表现出不同的人格，从而凸显了LLMs人格与人类人格之间的根本差异。通过我们的工作，我们呼吁重新评估LLMs中人格的定义与测量方式。
英文摘要：
As Large Language Models (LLMs) are integrated with human daily applications rapidly, many societal and ethical concerns are raised regarding the behavior of LLMs. One of the ways to comprehend LLMs' behavior is to analyze their personalities. Many recent studies quantify LLMs' personalities using self-assessment tests that are created for humans. Yet many critiques question the applicability and reliability of these self-assessment tests when applied to LLMs. In this paper, we investigate LLM personalities using an alternate personality measurement method, which we refer to as the external evaluation method, where instead of prompting LLMs with multiple-choice questions in the Likert scale, we evaluate LLMs' personalities by analyzing their responses toward open-ended situational questions using an external machine learning model. We first fine-tuned a Llama2-7B model as the MBTI personality predictor that outperforms the state-of-the-art models as the tool to analyze LLMs' responses. Then, we prompt the LLMs with situational questions and ask them to generate Twitter posts and comments, respectively, in order to assess their personalities when playing two different roles. Using the external personality evaluation method, we identify that the obtained personality types for LLMs are significantly different when generating posts versus comments, whereas humans show a consistent personality profile in these two different situations. This shows that LLMs can exhibit different personalities based on different scenarios, thus highlighting a fundamental difference between personality in LLMs and humans. With our work, we call for a re-evaluation of personality definition and measurement in LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
