大型语言模型在PTSD严重程度评估中的系统评估：上下文知识与建模策略的作用
A Systematic Evaluation of Large Language Models for PTSD Severity Estimation: The Role of Contextual Knowledge and Modeling Strategies
AI总结：
本研究系统评估了11种大型语言模型在PTSD严重程度评估中的表现，发现提供详细定义和上下文可提升准确性，增加推理努力可改善估计，并验证了集成策略的有效性。
摘要：
大型语言模型（LLMs）越来越多地被以零样本（生成式）方式用于评估心理健康状况，然而我们对影响其准确性的因素了解有限。在本研究中，我们使用来自1,437名个体的自然语言叙述和自我报告的PTSD严重程度评分的临床数据集，全面评估了11种最先进的LLMs的性能。为了理解影响模型评估准确性的因素，我们系统地变化了（i）提示给模型的上下文知识，如子量表定义、分布摘要和访谈问题，以及（ii）建模策略，包括零样本与少样本、推理努力量、模型大小、结构化子量表与直接标量预测、输出重新缩放和九种集成方法。我们的发现表明：（a）当提供详细的构念定义和叙述上下文时，LLMs最为准确，甚至超过人类评分者与自我报告评分的一致性；（b）增加推理努力导致更好的估计准确性；（c）开放权重模型（Llama, DeepSeek）的性能在超过700亿参数后趋于平稳，而封闭权重模型（gpt-o3-mini, gpt-5）随着新一代的推出而改进；（d）当将监督模型与零样本LLMs集成时，达到最佳性能。除了与自我报告的一致性外，LLMs的估计能够区分PTSD严重程度与抑郁、焦虑和酒精使用，并前瞻性地预测未来的精神医疗支出。总之，这些结果表明上下文知识和建模策略显著影响基于LLMs的PTSD严重程度评估的准确性和临床实用性。
英文摘要：
Large language models (LLMs) are increasingly being used in a zero-shot (generative) fashion to assess mental health conditions, yet we have limited knowledge on what factors affect their accuracy. In this study, we use a clinical dataset of natural language narratives and self-reported PTSD severity scores from 1,437 individuals to comprehensively evaluate the performance of 11 state-of-the-art LLMs. To understand the factors affecting model's assessment accuracy, we systematically varied (i) contextual knowledge prompted to the models like subscale definitions, distribution summary, and interview questions, and (ii) modeling strategies including zero-shot vs few shot, amount of reasoning effort, model sizes, structured subscales vs direct scalar prediction, output rescaling and nine ensemble methods. Our findings indicate that (a) LLMs are most accurate when provided with detailed construct definitions and context of the narrative, even exceeding human raters agreement with self-reported scores; (b) increased reasoning effort leads to better estimation accuracy; (c) performance of open-weight models (Llama, DeepSeek) plateaus beyond 70B parameters while closed-weight (gpt-o3-mini, gpt-5) alternatives improve with newer generations; and (d) best performance is achieved when ensembling a supervised model with the zero-shot LLMs. Beyond agreement with self-reports, LLMs' estimates discriminated PTSD severity from depression, anxiety, and alcohol use, and prospectively predicted future mental healthcare expenditure. Together, these results suggest that contextual knowledge and modeling strategies meaningfully affect accuracy and clinical utility of LLM-based assessments of PTSD severity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
