通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。
arXiv 2311.05297cs.CLcs.AIcs.LG
挑战大语言模型人格测试的有效性
Challenging the Validity of Personality Tests for Large Language Models
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Tübingen AI Center（蒂宾根人工智能中心）；ETH Zurich（苏黎世联邦理工学院）；Methods Center（方法中心）；University of Tübingen（蒂宾根大学）
Tom Sühr, Florian E. Dorner, Samira Samadi, Augustin Kelava
备注：
A less extensive and shorter version of this work has been accepted at Socially Responsible Language Modelling Research (SoLaR) 2023 Workshop at NeurIPS 2023
With large language models (LLMs) like GPT-4 appearing to behave increasingly human-like in text-based interactions, it has become popular to attempt to evaluate personality traits of LLMs using questionnaires originally developed for humans. While reusing measures is a resource-efficient way to evaluate LLMs, careful adaptations are usually required to ensure that assessment results are valid even across human subpopulations. In this work, we provide evidence that LLMs' responses to personality tests systematically deviate from human responses, implying that the results of these tests cannot be interpreted in the same way. Concretely, reverse-coded items ("I am introverted" vs. "I am extraverted") are often both answered affirmatively. Furthermore, variation across prompts designed to "steer" LLMs to simulate particular personality types does not follow the clear separation into five independent personality factors from human samples. In light of these results, we believe that it is important to investigate tests' validity for LLMs before drawing strong conclusions about potentially ill-defined concepts like LLMs' "personality".
Beyond BFI: The CSI for Enhanced Reliability and Validity in Evaluating LLM Personality Traits
University of Illinois Chicago（伊利诺伊大学芝加哥分校）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Microsoft Research Asia（微软亚洲研究院）；North Carolina State University（北卡罗来纳州立大学）
Generative Personality Simulation via Theory-Informed Structured Interview
Department of Psychological Sciences & 2 Department of Computer Science, Rice University（1 心理学系 & 2 计算机科学系，莱斯大学）；Department of Electrical and Computer Engineering, Northeastern University（3 电气与计算机工程系，东北大学）；Department of Computer Science, Johns Hopkins University（4 计算机科学系，约翰霍普金斯大学）；Microsoft Research Asia（微软亚洲研究院）
Automatic Item Generation for Personality Situational Judgment Tests with Large Language Models
Beijing Normal University（北京师范大学）；Central China Normal University（华中师范大学）；Key Laboratory of Adolescent Cyberpsychology and Behavior, Ministry of Education（教育部青少年网络心理与行为重点实验室）；Beijing Key Laboratory of Applied Experimental Psychology（北京市应用实验心理学重点实验室）；Research Center for Capacity Building in Educational Assessment and Evaluation (Beijing Higher Education Innovation Center for Philosophy and Social Sciences)（教育评价与评估能力建设研究中心（北京高等教育哲学社会科学创新中心））
Predicting the Big Five Personality Traits in Chinese Counselling Dialogues Using Large Language Models
Zhejiang University（浙江大学）；School of Engineering, Westlake University（西湖大学工程学院）；Westlake University（西湖大学）；School of Early Childhood Education, Hangzhou Polytechnic（杭州职业技术学院幼儿教育学院）
Scaling Law in LLM Simulated Personality: More Detailed and Realistic Persona Profile Is All You Need
Department of Computer；Information Engineering, Hebei Petroleum University of Technology, Chengde, China (e-mail；School of Computer Science \& Technology, Beijing Institute of Technology, Beijing, China (e-mail
Computational Phenomenology of Borderline Personality Disorder: A Comparative Evaluation of LLM-Simulated Expert Personas and Human Clinical Experts
IDEAS Research Institute（IDEAS研究机构）；Adam Mickiewicz University（亚当·密茨凯维奇大学）；AMU Center for Artificial Intelligence（AMU人工智能中心）；Poznań University of Medical Sciences（波兹南医科大学）；Maria Curie-Skłodowska University（玛丽·居里-斯洛多夫斯卡大学）；University of Warsaw（华沙大学）
BIG5-TPoT: Predicting BIG Five Personality Traits, Facets, and Items Through Targeted Preselection of Texts
The University of Maryland Applied Research Laboratory for Intelligence and Security (ARLIS)（马里兰大学应用研究实验室（情报与安全实验室））；The University of Maryland (UMD)（马里兰大学）
Split Personality Training: Revealing Latent Knowledge Through Alternate Personalities
University of Bergen, Bergen, Norway（卑尔根大学）；Independent Researcher, London, UK（独立研究者）；University College London, London, UK（伦敦大学学院）；Independent Researcher, Berlin, Germany（独立研究者）；Independent Researcher（独立研究者）；Head of Research（研发负责人）
Cognitive phantoms in LLMs through the lens of latent variables
Department of Methodology and Statistics（方法学与统计学系）；Tilburg University（蒂尔堡大学）；Department of Security and Crime Science（安全与犯罪科学系）；University College London（伦敦大学学院）
Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead
Max Planck Institute for Intelligent Systems, Tübingen（马克斯·普朗克智能系统研究所，图宾根）；ETH Zurich, Zurich（苏黎世联邦理工学院，苏黎世）；University of Tübingen, Methods Center, Tübingen（图宾根大学，方法中心，图宾根）；Massachusetts Institute of Technology, Cambridge（麻省理工学院，剑桥）
University of Oxford（牛津大学）；The Alan Turing Institute（艾伦·图灵研究所）；University of Warwick（华威大学）
Bipolar disorder (BD) and borderline personality disorder (BPD) are both chronic psychiatric disorders. However, their overlapping symptoms and common comorbidity make it challenging for the clinicians to distinguish the two conditions on the basis of a clinical interview. In this work, we first present a new multi-modal dataset containing interviews involving individuals with BD or BPD being interviewed about a non-clinical topic . We investigate the automatic detection of the two conditions, and demonstrate a good linear classifier that can be learnt using a down-selected set of features fro…
The use of GPT-4o and Other Large Language Models for the Improvement and Design of Self-Assessment Scales for Measurement of Interpersonal Communication Skills
University of Zagreb（萨格勒布大学）；Faculty of Organization and Informatics（组织与信息学院）
Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact
Max Planck Institute for Human Development（马克斯·普朗克人类发展研究所）；University of Konstanz（康斯坦茨大学）；Barcelona Supercomputing Center（巴塞罗那超级计算中心）；University of Basel（巴塞尔大学）
We present NLU++, a novel dataset for natural language understanding (NLU) in task-oriented dialogue (ToD) systems, with the aim to provide a much more challenging evaluation environment for dialogue NLU models, up to date with the current application and industry requirements. NLU++ is divided into two domains (BANKING and HOTELS) and brings several crucial improvements over current commonly used NLU datasets. 1) NLU++ provides fine-grained domain ontologies with a large set of challenging multi-intent sentences, introducing and validating the idea of intent modules that can be combined into …
