推理并非万能：检验大语言模型在多轮心理健康对话中的表现
Reasoning Is Not All You Need: Examining LLMs for Multi-Turn Mental Health Conversations
AI总结：
本文提出MedAgent框架生成多轮心理健康对话数据集MHSD，并构建MultiSenseEval评估框架，发现前沿推理模型在患者中心沟通和高级诊断上表现不佳，且性能随对话轮次增加而下降。
摘要：
心理健康医疗资源获取受限、等待时间漫长以及大语言模型（LLMs）能力的不断提升，促使人们转向LLMs来满足心理健康需求。然而，对LLMs多轮心理健康对话能力的考察仍鲜有研究。现有评估框架通常侧重于诊断准确性和胜率，往往忽视了实现有意义对话所需的与患者特定目标、价值观和人格特质的对齐。为解决这一问题，我们引入了MedAgent，这是一种用于合成生成真实多轮心理健康意义建构对话的新颖框架，并利用它创建了心理健康意义建构对话（MHSD）数据集，包含超过2,200次患者与LLM的对话。此外，我们提出了MultiSenseEval，这是一个以人为中心的标准来评估LLMs在医疗场景下多轮对话能力的整体框架。研究结果表明，前沿推理模型在以患者为中心的沟通方面表现不佳，在高级诊断能力上也存在困难，平均得分仅为31%。此外，我们观察到模型性能因患者画像不同而存在差异，且随着对话轮次增加性能出现下降。本研究提供了一个综合的合成数据生成框架、一个数据集以及一个评估框架，用于评估LLMs在多轮心理健康对话中的表现。
英文摘要：
Limited access to mental healthcare, extended wait times, and increasing capabilities of Large Language Models (LLMs) has led individuals to turn to LLMs for fulfilling their mental health needs. However, examining the multi-turn mental health conversation capabilities of LLMs remains under-explored. Existing evaluation frameworks typically focus on diagnostic accuracy and win-rates and often overlook alignment with patient-specific goals, values, and personalities required for meaningful conversations. To address this, we introduce MedAgent, a novel framework for synthetically generating realistic, multi-turn mental health sensemaking conversations and use it to create the Mental Health Sensemaking Dialogue (MHSD) dataset, comprising over 2,200 patient-LLM conversations. Additionally, we present MultiSenseEval, a holistic framework to evaluate the multi-turn conversation abilities of LLMs in healthcare settings using human-centric criteria. Our findings reveal that frontier reasoning models yield below-par performance for patient-centric communication and struggle at advanced diagnostic capabilities with average score of 31%. Additionally, we observed variation in model performance based on patient's persona and performance drop with increasing turns in the conversation. Our work provides a comprehensive synthetic data generation framework, a dataset and evaluation framework for assessing LLMs in multi-turn mental health conversations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
