arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.18443cs.HCcs.AIcs.CLcs.CY
收藏

从‘帮助’到‘有用’:对LLMs在心理健康电子服务中的分层评估

From "Help" to Helpful: A Hierarchical Assessment of LLMs in Mental e-Health Applications

Technische Hochschule Nürnberg – Georg Simon Ohm(纽伦堡技术大学——格奥尔格·西蒙·欧姆学院)

Philipp Steigerwald, Jens Albrecht

AI总结:

本研究评估了11种LLM在生成德国咨询邮件主题行中的表现,发现德国微调能提升性能,同时探讨了心理健康AI部署中的隐私、偏见和问责制等伦理问题。

摘要:

心理社会在线咨询经常遇到通用的主题行,这阻碍了有效案例优先级的确定。本研究评估了11个大型语言模型生成六个词的主题行用于德国咨询电子邮件,通过分层评估——首先对输出进行分类,然后在类别内排序以实现可管理的评估。九位评估者(心理咨询专业人士和AI系统)通过Krippendorff的α、Spearman的ρ、Pearson的r和Kendall的τ进行分析。结果揭示了专有服务与隐私保护的开源替代品之间的性能权衡,德国微调始终提高了性能。本研究针对心理健康AI部署中的关键伦理问题,包括隐私、偏见和问责制。

英文摘要:

Psychosocial online counselling frequently encounters generic subject lines that impede efficient case prioritisation. This study evaluates eleven large language models generating six-word subject lines for German counselling emails through hierarchical assessment - first categorising outputs, then ranking within categories to enable manageable evaluation. Nine assessors (counselling professionals and AI systems) enable analysis via Krippendorff's $α$, Spearman's $ρ$, Pearson's $r$ and Kendall's $τ$. Results reveal performance trade-offs between proprietary services and privacy-preserving open-source alternatives, with German fine-tuning consistently improving performance. The study addresses critical ethical considerations for mental health AI deployment including privacy, bias and accountability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑