ChatGPT能否评估人类性格？一个通用评估框架
Can ChatGPT Assess Human Personalities? A General Evaluation Framework
AI总结：
本文提出基于MBTI测试的通用框架，使ChatGPT等LLM能评估人类性格，并通过三个指标验证其评估的一致性与公平性。
摘要：
大型语言模型（LLMs），尤其是ChatGPT，已在多个领域产生了令人印象深刻的结果，但其潜在的人类化心理学特性在很大程度上仍未得到探索。现有工作研究了LLMs的虚拟性格，但很少探索通过LLMs分析人类性格的可能性。本文提出了一个基于迈尔斯-布里格斯类型指标（MBTI）测试的通用评估框架，用于让LLMs评估人类性格。具体而言，我们首先通过随机排列MBTI问题中的选项来设计无偏提示，并采用平均测试结果以鼓励更公正的答案生成。然后，我们提出替换问题陈述中的主语，使LLMs能够对不同主体进行灵活的查询和评估。最后，我们以正确性评估的方式重新表述问题指令，以促进LLMs生成更清晰的响应。所提出的框架使LLMs能够灵活评估不同人群的性格。我们进一步提出了三个评估指标，用于衡量包括ChatGPT和GPT-4在内的最先进LLMs评估结果的一致性、鲁棒性和公平性。我们的实验揭示了ChatGPT评估人类性格的能力，平均结果表明，尽管与InstructGPT相比，其对提示偏差的鲁棒性较低，但ChatGPT能够实现更一致和更公平的评估。
英文摘要：
Large Language Models (LLMs) especially ChatGPT have produced impressive results in various areas, but their potential human-like psychology is still largely unexplored. Existing works study the virtual personalities of LLMs but rarely explore the possibility of analyzing human personalities via LLMs. This paper presents a generic evaluation framework for LLMs to assess human personalities based on Myers Briggs Type Indicator (MBTI) tests. Specifically, we first devise unbiased prompts by randomly permuting options in MBTI questions and adopt the average testing result to encourage more impartial answer generation. Then, we propose to replace the subject in question statements to enable flexible queries and assessments on different subjects from LLMs. Finally, we re-formulate the question instructions in a manner of correctness evaluation to facilitate LLMs to generate clearer responses. The proposed framework enables LLMs to flexibly assess personalities of different groups of people. We further propose three evaluation metrics to measure the consistency, robustness, and fairness of assessment results from state-of-the-art LLMs including ChatGPT and GPT-4. Our experiments reveal ChatGPT's ability to assess human personalities, and the average results demonstrate that it can achieve more consistent and fairer assessments in spite of lower robustness against prompt biases compared with InstructGPT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
