A Novel Nuanced Conversation Evaluation Framework for Large Language Models in Mental Health
School of Computer Science, Faculty of Engineering（工程学院计算机科学学院）；The University of Sydney（悉尼大学）；School of Computer Science and Engineering（计算机科学与工程学院）；University of New South Wales（新南威尔士大学）；School of Optometry and Vision Science（视光与视觉科学学院）
Alexander Marrapese, Basem Suleiman, Imdad Ullah, Juno Kim
Understanding the conversation abilities of Large Language Models (LLMs) can help lead to its more cautious and appropriate deployment. This is especially important for safety-critical domains like mental health, where someone's life may depend on the exact wording of a response to an urgent question. In this paper, we propose a novel framework for evaluating the nuanced conversation abilities of LLMs. Within it, we develop a series of quantitative metrics developed from literature on using psychotherapy conversation analysis literature. While we ensure that our framework and metrics are transferable by researchers to relevant adjacent domains, we apply them to the mental health field. We use our framework to evaluate several popular frontier LLMs, including some GPT and Llama models, through a verified mental health dataset. Our results show that GPT4 Turbo can perform significantly more similarly to verified therapists than other selected LLMs. We conduct additional analysis to examine how LLM conversation performance varies across specific mental health topics. Our results indicate that GPT4 Turbo performs well in achieving high correlation with verified therapists in particular topics such as Parenting and Relationships. We believe our contributions will help researchers develop better LLMs that, in turn, will more positively support people's lives.
MDD-5k: A New Diagnostic Conversation Dataset for Mental Disorders Synthesized via Neuro-Symbolic LLM Agents
Theta Health Inc.（西塔健康公司）；Chen Frontier Lab for AI and Mental Health, Tianqiao and Chrissy Chen Institute（天桥脑科学研究院陈天桥雒芊芊脑科学研究院AI与精神健康前沿实验室）；Shanghai Mental Health Center（上海市精神卫生中心）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）；Shanghai Clinical Research Center for Mental Health（上海市精神疾病临床医学中心）；Shanghai Key Laboratory of Psychotic Disorders（上海市精神疾病重点实验室）
Detecting Conversational Mental Manipulation with Intent-Aware Prompting
The University of Sydney（悉尼大学）；University of Technology Sydney（悉尼科技大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Harvard University（哈佛大学）；University of New South Wales（新南威尔士大学）
K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations
University of Roehampton（罗汉普顿大学）；Kivira Health；University of Hertfordshire（赫特福德大学）；University of Surrey（萨里大学）；University of Bedfordshire（贝德福德大学）；Tavistock Relationships（塔维斯托克关系中心）；InsideOut
SELF-PERCEPT: Introspection Improves Large Language Models' Detection of Multi-Person Mental Manipulation in Conversations
Manipal University Jaipur, India（马普尔大学斋普尔，印度）；Manipal Institute of Technology, India（马普尔理工学院，印度）；AryaXAI Alignment Lab, AryaXAI.com, India（AryaXAI对齐实验室，AryaXAI.com，印度）；National Institute of Technology Karnataka, Surathkal, India（卡纳塔克国家理工学院，苏拉塔卡尔，印度）；IISER Kolkata, India（印度科钦IISER）
MoPHES:Leveraging on-device LLMs as Agent for Mobile Psychological Health Evaluation and Support
School of Software Engineering, Jiangxi University of Science and Technology（江西科技理工大学软件工程学院）；School of Computer Science, Shenzhen University（深圳大学计算机科学学院）
Auditing Framing-Sensitive Behavioral Instability in Large Language Models for Mental Health Interactions
a School of computer science, digital engineering and AI, Long Island University, Brooklyn, NY, USA（a 计算科学与数字工程及人工智能学院，罗格斯大学，布鲁克林，纽约，美国）；b Department of Psychology, Long Island University, Brooklyn, NY, USA（b 心理学系，罗格斯大学，布鲁克林，纽约，美国）
Decoding Workload and Agreement From EEG During Spoken Dialogue With Conversational AI
Chair of Neuroadaptive Human--Computer Interaction（神经适应性人机交互教授席位）；Brandenburg Technical University Cottbus--Senftenberg（勃兰登堡技术大学库滕堡-森芬根堡分校）；Auryal GmbH（Auryal公司）；Cottbus, Germany（库滕堡，德国）
MAQuA: Adaptive Question-Asking for Multidimensional Mental Health Screening using Item Response Theory
Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）；Stony Brook University（石溪大学）；Aarhus University（奥胡斯大学）；Department of Psychology, Lund University（吕勒欧大学心理学系）；College of Connected Computing, Vanderbilt University（范德比尔特大学连接计算学院）
Dynamic Demonstration Retrieval and Cognitive Understanding for Emotional Support Conversation
Sun Yat-sen University（中山大学）；Alibaba group（阿里巴巴集团）；Guangdong Provincial Key Laboratory of Fire Science and Intelligent Emergency Technology（广东省火灾科学与智能应急技术重点实验室）；Pazhou Lab（琶洲实验室）
Large Language Model-Powered Conversational Agent Delivering Problem-Solving Therapy (PST) for Family Caregivers: Enhancing Empathy and Therapeutic Alliance Using In-Context Learning
Florida State University（佛罗里达州立大学）；University of Washington（华盛顿大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
CARE: Counselor-Aligned Response Engine for Online Mental-Health Support
Ben-Gurion University（本· Gurion 大学）；School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；School of Informatics, University of Edinburgh Edinburgh UK（爱丁堡大学信息学院爱丁堡 英国）
