arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2505.03427cs.CLcs.AIcs.HC
收藏

MedArabiQ:在阿拉伯语医学任务上对大型语言模型进行基准评测

MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks

New York University Abu Dhabi(纽约大学阿布扎克分校)

Mouath Abu Daoud, Chaimae Abouzahir, Leen Kharouf, Walid Al-Eisawi, Nizar Habash, Farah E. Shamout

AI总结:

本文提出阿拉伯语医学基准MedArabiQ,涵盖七个任务并评估GPT-4o等五个主流LLM,揭示多语言医疗基准的缺乏,为提升LLM在医疗中的公平多语言能力奠定基础。

摘要:

大型语言模型(LLMs)已在医疗保健的多种应用中展现出显著潜力。然而,由于缺乏高质量的领域特定数据集和基准,它们在阿拉伯语医学领域的有效性仍未得到探索。本研究引入了MedArabiQ,这是一个新颖的基准数据集,包含七个阿拉伯语医学任务,覆盖多个医学专科,并包括多项选择题、填空和患者-医生问答。我们首先利用过往医学考试和公开可用数据集构建了该数据集。随后,我们引入了不同的修改以评估各种LLM能力,包括偏见缓解。我们使用五个最先进的开源和专有LLM进行了广泛评估,包括GPT-4o、Claude 3.5-Sonnet和Gemini 1.5。我们的研究结果强调,需要创建跨越不同语言的高质量新基准,以确保LLM在医疗保健中的公平部署和可扩展性。通过建立这一基准并发布数据集,我们为未来旨在评估和增强LLM多语言能力的研究奠定了基础,以促进生成式AI在医疗保健中的公平使用。

英文摘要:

Large Language Models (LLMs) have demonstrated significant promise for various applications in healthcare. However, their efficacy in the Arabic medical domain remains unexplored due to the lack of high-quality domain-specific datasets and benchmarks. This study introduces MedArabiQ, a novel benchmark dataset consisting of seven Arabic medical tasks, covering multiple specialties and including multiple choice questions, fill-in-the-blank, and patient-doctor question answering. We first constructed the dataset using past medical exams and publicly available datasets. We then introduced different modifications to evaluate various LLM capabilities, including bias mitigation. We conducted an extensive evaluation with five state-of-the-art open-source and proprietary LLMs, including GPT-4o, Claude 3.5-Sonnet, and Gemini 1.5. Our findings highlight the need for the creation of new high-quality benchmarks that span different languages to ensure fair deployment and scalability of LLMs in healthcare. By establishing this benchmark and releasing the dataset, we provide a foundation for future research aimed at evaluating and enhancing the multilingual capabilities of LLMs for the equitable use of generative AI in healthcare.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑