arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.15966cs.CLcs.AI
收藏

评估基础模型在精神卫生保健问答分类中的有效性

Evaluating the Effectiveness of the Foundational Models for Q&A Classification in Mental Health care

Umm Al-Qura University(乌姆阿尔古拉大学);King Khalid University(国王哈立德大学)

Hassan Alhuzali, Ashwag Alasmari

AI总结:

本研究针对阿拉伯语精神卫生领域问答分类任务,评估四类学习方法的性能,发现微调PLMs及GPT-3.5少样本提示均有优异表现,为阿拉伯语精神卫生支持技术应用提供依据。

摘要:

预训练语言模型(Pre-trained Language Models, PLMs)有望通过提供可及且具备文化敏感性的资源,变革精神卫生支持服务。但尽管具备这一潜力,其在精神卫生保健领域、尤其是针对阿拉伯语的有效性尚未得到广泛探索。为填补这一空白,本研究评估了基础模型在精神卫生保健领域问答(Questions and Answers, Q&A)分类任务中的有效性。我们使用了MentalQA数据集,这是一个包含精神卫生相关问答交互内容的阿拉伯语数据集。本研究采用四种不同的学习方法开展实验:传统特征提取法、作为特征提取器的PLMs、PLMs微调,以及在零样本和少样本学习设置下对大语言模型(GPT-3.5和GPT-4)进行提示工程。传统特征提取器结合支持向量机(Support Vector Machines, SVM)已展现出可观性能,但PLMs凭借捕捉语义的能力取得了更优结果。例如,MARBERT取得了最高性能,问题分类的杰卡德系数(Jaccard Score)为0.80,答案分类的杰卡德系数为0.86。我们进一步开展了深度分析,包括对比微调与非微调的效果、探究数据规模变化的影响,以及进行错误分析。分析表明,微调对提升PLMs性能具有积极作用,训练数据规模对实现高性能至关重要。我们还探索了提示工程方法,其中GPT-3.5的少样本学习取得了可观结果,问题分类性能提升12%,答案分类性能提升45%。基于研究结果可以得出结论:PLMs和基于提示的方法在阿拉伯语精神卫生支持领域具有应用前景。

英文摘要:

Pre-trained Language Models (PLMs) have the potential to transform mental health support by providing accessible and culturally sensitive resources. However, despite this potential, their effectiveness in mental health care and specifically for the Arabic language has not been extensively explored. To bridge this gap, this study evaluates the effectiveness of foundational models for classification of Questions and Answers (Q&A) in the domain of mental health care. We leverage the MentalQA dataset, an Arabic collection featuring Q&A interactions related to mental health. In this study, we conducted experiments using four different types of learning approaches: traditional feature extraction, PLMs as feature extractors, Fine-tuning PLMs and prompting large language models (GPT-3.5 and GPT-4) in zero-shot and few-shot learning settings. While traditional feature extractors combined with Support Vector Machines (SVM) showed promising performance, PLMs exhibited even better results due to their ability to capture semantic meaning. For example, MARBERT achieved the highest performance with a Jaccard Score of 0.80 for question classification and a Jaccard Score of 0.86 for answer classification. We further conducted an in-depth analysis including examining the effects of fine-tuning versus non-fine-tuning, the impact of varying data size, and conducting error analysis. Our analysis demonstrates that fine-tuning proved to be beneficial for enhancing the performance of PLMs, and the size of the training data played a crucial role in achieving high performance. We also explored prompting, where few-shot learning with GPT-3.5 yielded promising results. There was an improvement of 12% for question and classification and 45% for answer classification. Based on our findings, it can be concluded that PLMs and prompt-based approaches hold promise for mental health support in Arabic.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑