PsychiatryBench: 一个面向心理治疗的多任务基准测试
PsychiatryBench: A Multi-Task Benchmark for LLMs in Psychiatry
AI总结：
PsychiatryBench通过权威教科书和病例书构建多任务基准，评估LLMs在心理治疗中的临床一致性与安全性，揭示模型在多轮随访等任务中的不足，推动专门调优和更严谨的评估方法。
摘要：
大型语言模型（LLMs）在提升 psychiatric 实践中具有巨大潜力，从提高诊断准确性到简化临床文档和治疗支持。然而，现有的评估资源严重依赖于小型临床访谈语料、社交媒体帖子或合成对话，这限制了其临床有效性，并未能捕捉到诊断推理的全部复杂性。在本工作中，我们介绍了PsychiatryBench，一个严格筛选的基准测试，其仅基于权威的、由专家验证的心理学教科书和病例书。PsychiatryBench包含十一项不同的问答任务，从诊断推理和治疗计划到长期随访、管理计划、临床方法、连续病例分析以及多选/扩展匹配格式，总计5188项专家标注项目。我们评估了一组前沿LLMs（包括Google Gemini、DeepSeek、Sonnet 4.5和GPT 5）以及领先的开源医学模型如MedGemma，使用传统指标和
英文摘要：
Large language models (LLMs) offer significant potential in enhancing psychiatric practice, from improving diagnostic accuracy to streamlining clinical documentation and therapeutic support. However, existing evaluation resources heavily rely on small clinical interview corpora, social media posts, or synthetic dialogues, which limits their clinical validity and fails to capture the full complexity of diagnostic reasoning. In this work, we introduce PsychiatryBench, a rigorously curated benchmark grounded exclusively in authoritative, expert-validated psychiatric textbooks and casebooks. PsychiatryBench comprises eleven distinct question-answering tasks ranging from diagnostic reasoning and treatment planning to longitudinal follow-up, management planning, clinical approach, sequential case analysis, and multiple-choice/extended matching formats totaling 5,188 expert-annotated items. {\color{red}We evaluate a diverse set of frontier LLMs (including Google Gemini, DeepSeek, Sonnet 4.5, and GPT 5) alongside leading open-source medical models such as MedGemma using both conventional metrics and an "LLM-as-judge" similarity scoring framework. Our results reveal substantial gaps in clinical consistency and safety, particularly in multi-turn follow-up and management tasks, underscoring the need for specialized model tuning and more robust evaluation paradigms. PsychiatryBench offers a modular, extensible platform for benchmarking and improving LLM performance in mental health applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
