LLM 能否“自我报告”？：评估自我报告量表在衡量基于LLM的聊天机器人个性设计中的有效性
Can LLM "Self-report"?: Evaluating the Validity of Self-report Scales in Measuring Personality Design in LLM-based Chatbots
AI总结：
本研究通过创建500个不同个性设计的聊天机器人，评估自我报告量表在衡量LLM聊天机器人个性中的有效性，发现其与人类感知及交互质量弱相关，质疑了该方法效度并提出改进方向。
摘要：
聊天机器人的个性设计是交互质量的关键。随着聊天机器人从基于规则的系统演变为由大语言模型（LLM）驱动的系统，评估其个性设计有效性的难度日益增加，这尤其源于交互的开放性。近期一种被广泛采用的方法是利用自我报告问卷来评估基于LLM的聊天机器人的个性设计。这些问卷通常借鉴自成熟的人类个性量表，要求聊天机器人就各种个性特质对自己进行评分。基于LLM的聊天机器人能否有意义地“自我报告”其个性？我们创建了500个具有不同个性设计的聊天机器人，并通过考察人类在与这些机器人交互过程中形成的感知，评估了其自我报告个性分数的有效性。我们的研究结果表明，聊天机器人在人类个性量表上的回答与人类感知的个性特质及整体交互质量均仅呈现弱相关。这些发现对自我报告方法在此情境下的效标效度和预测效度提出了质疑。进一步的分析揭示了任务情境和交互在聊天机器人个性设计评估中的作用。我们进一步讨论了创建更具情境化和交互性评估的设计启示。
英文摘要：
A chatbot's personality design is key to interaction quality. As chatbots evolved from rule-based systems to those powered by large language models (LLMs), evaluating the effectiveness of their personality design has become increasingly complex, particularly due to the open-ended nature of interactions. A recent and widely adopted method for assessing the personality design of LLM-based chatbots is the use of self-report questionnaires. These questionnaires, often borrowed from established human personality inventories, ask the chatbot to rate itself on various personality traits. Can LLM-based chatbots meaningfully "self-report" their personality? We created 500 chatbots with distinct personality designs and evaluated the validity of their self-report personality scores by examining human perceptions formed during interactions with these chatbots. Our findings indicate that the chatbot's answers on human personality scales exhibit weak correlations with both human-perceived personality traits and the overall interaction quality. These findings raise concerns about both the criterion validity and the predictive validity of self-report methods in this context. Further analysis revealed the role of task context and interaction in the chatbot's personality design assessment. We further discuss design implications for creating more contextualized and interactive evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
