MentalBench: 一个用于评估大语言模型 psychiatric 诊断能力的 DSM 基础基准
MentalBench: A DSM-Grounded Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models
AI总结：
本文提出 MentalBench，一个用于评估大语言模型在不同临床模糊程度下能否做出 DSM 基础的 psychiatric 诊断决策的基准。该基准基于 psychiatrist 构建并验证的知识图谱，生成了 24,750 个合成临床案例，以系统地变化信息完整性和诊断复杂性，从而实现 DSM 基础的评估。实验表明，尽管最先进的 LLM 在噪声自由查询上表现良好，但它们在区分具有重叠症状的诊断时难以校准其信心。
摘要：
大型语言模型 (LLMs) 已吸引越来越多的关注，作为心理评估和临床决策支持的支持工具。然而，现有的心理健康基准大多依赖于社交媒体数据或支持性对话设置，限制了它们评估模型是否能够应用正式诊断标准和鉴别诊断规则的能力。在本文中，我们介绍了 MentalBench，一个用于评估 LLM 是否能在不同水平的临床模糊性下做出 DSM 基础的 psychiatric 诊断决策的基准。MentalBench 的核心是 MentalKG，一个由精神科医生构建并验证的知识图谱，编码了 DSM-5 的诊断标准和鉴别诊断规则，适用于 23 种心理疾病。利用 MentalKG 作为专家整理的逻辑基础，我们生成了 24,750 个合成临床案例，这些案例在信息完整性和诊断复杂性方面系统地变化，从而实现 DSM 基础的评估。我们的实验表明，尽管最先进的 LLM 在噪声自由查询上表现良好，但它们在区分具有重叠症状的诊断时难以校准其信心。这些发现引发了关于 LLM 作为心理决策支持工具可靠性的担忧，并突显了需要更多评估以反映现实世界心理诊断中的多样化挑战的必要性。
英文摘要：
Large language models (LLMs) have attracted growing interest as supportive tools for psychiatric assessment and clinical decision support. However, existing mental health benchmarks largely rely on social media data or supportive dialogue settings, limiting their ability to assess whether models can apply formal diagnostic criteria and differential diagnostic rules. In this paper, we introduce MentalBench, a benchmark for evaluating whether LLMs can make DSM-grounded psychiatric diagnostic decisions under varying levels of clinical ambiguity. At the core of MentalBench is MentalKG, a psychiatrist-built and validated knowledge graph encoding DSM-5 diagnostic criteria and differential diagnostic rules for 23 psychiatric disorders. Using MentalKG as an expert-curated logical backbone, we generate 24,750 synthetic clinical cases that systematically vary in information completeness and diagnostic complexity, enabling DSM-grounded evaluation. Our experiments show that although state-of-the-art LLMs perform well on noise-free queries that probe DSM-5 knowledge, they struggle to calibrate their confidence when distinguishing between disorders with overlapping symptoms. These findings raise concerns about the reliability of LLMs as psychiatric decision-support tools and highlight the need for more evaluation that reflects the diverse challenges in real-world psychiatric diagnosis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
