Mental health in children and adolescents has been steadily deteriorating over the past few years. The recent advent of Large Language Models (LLMs) offers much hope for cost and time efficient scaling of monitoring and intervention, yet despite specifically prevalent issues such as school bullying and eating disorders, previous studies on have not investigated performance in this domain or for open information extraction where the set of answers is not predetermined. We create a new dataset of Reddit posts from adolescents aged 12-19 annotated by expert psychiatrists for the following categories: TRAUMA, PRECARITY, CONDITION, SYMPTOMS, SUICIDALITY and TREATMENT and compare expert labels to annotations from two top performing LLMs (GPT3.5 and GPT4). In addition, we create two synthetic datasets to assess whether LLMs perform better when annotating data as they generate it. We find GPT4 to be on par with human inter-annotator agreement and performance on synthetic data to be substantially higher, however we find the model still occasionally errs on issues of negation and factuality and higher performance on synthetic data is driven by greater complexity of real data rather than inherent advantage.
Spontaneous Speech-Based Suicide Risk Detection Using Whisper and Large Language Models
Tsinghua University(清华大学);Vanke School of Public Health, Tsinghua University(清华大学万科公共卫生学院);Institute for Healthy China, Tsinghua University(清华大学健康中国研究院);Cambridge University Engineering Department(剑桥大学工程系)
Identifying Products in Online Cybercrime Marketplaces: A Dataset for Fine-grained Domain Adaptation
UT Austin(德克萨斯大学奥斯汀分校);University of Michigan(密歇根大学);Carnegie Mellon University(卡内基梅隆大学);UC Berkeley(加州大学伯克利分校);ICSI(国际计算机科学研究所);NYU(纽约大学);UC San Diego(加州大学圣地亚哥分校)