arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.16461cs.CL
收藏

大型语言模型在识别青少年在线论坛心理健康因素方面表现与专家相当

Large Language Models Perform on Par with Experts Identifying Mental Health Factors in Adolescent Online Forums

University of Oxford(牛津大学);University of Liverpool(利物浦大学)

Isabelle Lorge, Dan W. Joyce, Andrey Kormilitzin

AI总结:

本研究构建了青少年Reddit帖子数据集,对比专家与GPT3.5、GPT4在心理健康因素标注上的表现,发现GPT4与专家一致性相当,但在否定和事实上仍有局限。

摘要:

过去几年中,儿童和青少年的心理健康状况一直在持续恶化。大型语言模型(LLMs)的出现为高效扩展监控和干预带来了希望,尽管校园欺凌和饮食失调等问题尤为普遍,但以往的研究并未调查该领域的表现,也未涉及答案集未预先确定的开放信息提取。我们创建了一个包含12至19岁青少年Reddit帖子的新数据集,由精神科专家对以下类别进行标注:TRAUMA、PRECARITY、CONDITION、SYMPTOMS、SUICIDALITY和TREATMENT,并将专家标签与两个表现最佳的LLMs(GPT3.5和GPT4)的标注进行比较。此外,我们创建了两个合成数据集,以评估LLMs在生成数据的同时进行标注时是否表现更好。我们发现GPT4与人类标注者间的一致性相当,在合成数据上的表现显著更高,但模型在否定和事实性问题上仍偶尔出错,且在合成数据上更高的表现是由真实数据更大的复杂性驱动的,而非固有的优势。

英文摘要:

Mental health in children and adolescents has been steadily deteriorating over the past few years. The recent advent of Large Language Models (LLMs) offers much hope for cost and time efficient scaling of monitoring and intervention, yet despite specifically prevalent issues such as school bullying and eating disorders, previous studies on have not investigated performance in this domain or for open information extraction where the set of answers is not predetermined. We create a new dataset of Reddit posts from adolescents aged 12-19 annotated by expert psychiatrists for the following categories: TRAUMA, PRECARITY, CONDITION, SYMPTOMS, SUICIDALITY and TREATMENT and compare expert labels to annotations from two top performing LLMs (GPT3.5 and GPT4). In addition, we create two synthetic datasets to assess whether LLMs perform better when annotating data as they generate it. We find GPT4 to be on par with human inter-annotator agreement and performance on synthetic data to be substantially higher, however we find the model still occasionally errs on issues of negation and factuality and higher performance on synthetic data is driven by greater complexity of real data rather than inherent advantage.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑