arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2508.05830cs.CLcs.CY
收藏

“镜像”大语言模型抑郁评估存在标准污染

"Mirror" Large Language Model Evaluations of Depression are Criterion Contaminated

Washington University in St. Louis(圣路易斯华盛顿大学);SMU(南卫理公会大学)

Tong Li, Rasiq Hussain, Mehak Gupta, Joshua R. Oltmanns

AI总结:

本研究揭示LLM抑郁评估中的“镜像”方法存在标准污染,提出非镜像评估更有效,为临床抑郁检测提供更可靠途径。

摘要:

使用抑郁评估中引发的语言反应来预测同一评估分数的大语言模型(LLM)研究,通常报告对抑郁的预测近乎完美。我们称这些为“镜像”评估,并展示了一个标准污染的应用案例。N=110名参与者完成了结构化诊断性抑郁访谈(镜像条件)和生活史访谈(非镜像条件)。LLM被提示在每种条件下预测抑郁分数。正如预期,镜像评估近乎完美。然而,非镜像评估也显示出在心理学中被认为出色的预测规模。此外,镜像和非镜像预测与患者健康问卷-9分数的相关性大小相似,这表明当预测独立的抑郁测量时,镜像条件的优势消失。主题建模揭示了不同访谈类型中抑郁相关主题的差异。镜像评估更应被视为可靠性评估而非有效性评估。将非镜像方法纳入LLM抑郁评估可能支持更有效和临床相关的应用。关键词:大语言模型,心理评估,精神病理学,抑郁,可靠性,有效性,标准污染。

英文摘要:

Large Language Model (LLM) studies that use language responses elicited from depression assessments to predict scores on those same assessments often report near-perfect prediction of depression. We refer to these as "Mirror" evaluations and demonstrate an applied case of criterion contamination. N = 110 participants completed both structured diagnostic depression interviews (Mirror condition) and life history interviews ("Non-Mirror" condition). LLMs were prompted to predict depression scores in each condition. As expected, Mirror evaluations were near-perfect. However, Non-Mirror evaluations also displayed prediction sizes considered outstanding in psychology. Further, both Mirror and Non-Mirror predictions correlated with Patient Health Questionnaire-9 scores at similar sizes, suggesting the Mirror condition's advantage collapses when predicting an independent depression measurement. Topic modeling revealed differing depression-related themes across interview types. Mirror evaluations are better considered as reliability evaluations than as validity evaluations. Incorporating Non-Mirror approaches in LLM depression assessment may support more valid and clinically-relevant applications. Keywords: large language models, psychological assessment, psychopathology, depression, reliability, validity, criterion contamination

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑