arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2507.08031cs.CL
收藏

超越规模:小型语言模型在心理健康理解方面可与 GPT-4 媲美

Beyond Scale: Small Language Models are Comparable to GPT-4 in Mental Health Understanding

University of Melbourne(墨尔本大学);University of Auckland(奥克兰大学);RMIT University(皇家墨尔本理工学院)

Hong Jia, Shiya Fu, Feng Xia, Vassilis Kostakos, Ting Dang

AI总结:

本文系统评估了小型语言模型在六项心理健康理解任务上的表现,发现其零样本性能与大型模型相差不到2%,少样本提示可带来高达14.6%的提升,证明了其作为隐私保护型可扩展心理健康筛查工具的潜力。

摘要:

小型语言模型(SLMs)作为隐私保护替代方案在敏感应用中的出现,引发了一个关于其与大型语言模型(LLMs)相比固有理解能力的基本问题。本文通过在多种分类任务上的系统评估,研究了当前 SLMs 的心理健康理解能力。采用零样本和少样本学习范式,我们将它们的性能与已建立的 LLM 基线进行基准比较,以阐明它们在这一关键领域的相对优势和局限性。我们在六项心理健康理解任务上评估了五种最先进的 SLMs(Phi-3、Phi-3.5、Qwen2.5、Llama-3.2、Gemma2)与三种 LLMs(GPT-4、FLAN-T5-XXL、Alpaca-7B)。我们的发现表明,在二分类任务中,SLMs 的平均性能与 LLMs 相差不到 2%(在零样本设置中 F1 分数为 0.64 对 0.66),证明尽管参数数量少了几个数量级,其仍具有显著能力。两类模型在多类严重程度任务上都经历了类似的性能下降(下降超过 30%),这表明细致的临床理解挑战超越了模型规模。少样本提示为 SLMs 提供了实质性改进(高达 14.6%),而 LLM 的增益则更不稳定。我们的工作突出了 SLMs 在心理健康理解方面的潜力,表明它们可以成为分析敏感在线文本数据的有效隐私保护工具。特别是,它们通过少样本学习以最少数据快速适应和专门化的能力,使其成为可扩展心理健康筛查工具的有希望的候选者。

英文摘要:

The emergence of Small Language Models (SLMs) as privacy-preserving alternatives for sensitive applications raises a fundamental question about their inherent understanding capabilities compared to Large Language Models (LLMs). This paper investigates the mental health understanding capabilities of current SLMs through systematic evaluation across diverse classification tasks. Employing zero-shot and few-shot learning paradigms, we benchmark their performance against established LLM baselines to elucidate their relative strengths and limitations in this critical domain. We assess five state-of-the-art SLMs (Phi-3, Phi-3.5, Qwen2.5, Llama-3.2, Gemma2) against three LLMs (GPT-4, FLAN-T5-XXL, Alpaca-7B) on six mental health understanding tasks. Our findings reveal that SLMs achieve mean performance within 2\% of LLMs on binary classification tasks (F1 scores of 0.64 vs 0.66 in zero-shot settings), demonstrating notable competence despite orders of magnitude fewer parameters. Both model categories experience similar degradation on multi-class severity tasks (a drop of over 30\%), suggesting that nuanced clinical understanding challenges transcend model scale. Few-shot prompting provides substantial improvements for SLMs (up to 14.6\%), while LLM gains are more variable. Our work highlights the potential of SLMs in mental health understanding, showing they can be effective privacy-preserving tools for analyzing sensitive online text data. In particular, their ability to quickly adapt and specialize with minimal data through few-shot learning positions them as promising candidates for scalable mental health screening tools.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑