Large Language Models (LLMs) have demonstrated significant promise for various applications in healthcare. However, their efficacy in the Arabic medical domain remains unexplored due to the lack of high-quality domain-specific datasets and benchmarks. This study introduces MedArabiQ, a novel benchmark dataset consisting of seven Arabic medical tasks, covering multiple specialties and including multiple choice questions, fill-in-the-blank, and patient-doctor question answering. We first constructed the dataset using past medical exams and publicly available datasets. We then introduced different modifications to evaluate various LLM capabilities, including bias mitigation. We conducted an extensive evaluation with five state-of-the-art open-source and proprietary LLMs, including GPT-4o, Claude 3.5-Sonnet, and Gemini 1.5. Our findings highlight the need for the creation of new high-quality benchmarks that span different languages to ensure fair deployment and scalability of LLMs in healthcare. By establishing this benchmark and releasing the dataset, we provide a foundation for future research aimed at evaluating and enhancing the multilingual capabilities of LLMs for the equitable use of generative AI in healthcare.
!MSA at AraHealthQA 2025 Shared Task: Enhancing LLM Performance for Arabic Clinical Question Answering through Prompt Engineering and Ensemble Learning
Faculty of Computer Science, MSA University(计算机科学学院,MSA大学)
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);New York University Abu Dhabi(纽约大学阿布扎比分校);IBM Research AI(IBM研究院人工智能部门);Qassim University(卡西姆大学)
AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data
Department of Information Systems, Faculty of Computing and Information Technology(信息系统系,计算与信息科技学院);King Abdulaziz University(国王阿卜杜勒阿齐兹大学);Department of Information Systems and Technology, College of Computer Science and Engineering(信息系统与技术系,计算机科学与工程学院);University of Jeddah(朱德赫大学);School of Computing and Creative Technologies(计算与创意科技学院);University of the West of England(西英格兰大学)
AraHealthQA 2025: The First Shared Task on Arabic Health Question Answering
Umm Al-Qura University(乌姆·阿勒·库拉大学);New York University Abu Dhabi(纽约大学阿布扎比分校);The University of British Columbia(不列颠哥伦比亚大学);King Khalid University(国王 Khalid 大学)
AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP
King Fahd University of Petroleum and Minerals(国王法赫德石油和矿物大学);SDAIA–KFUPM Joint Research Center for Artificial Intelligence(SDAIA-KFUPM人工智能联合研究中心);Mohammed VI Polytechnic University(穆莱·易卜拉欣(polytechnic)大学)
CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understanding
Instituto Politécnico Nacional(墨西哥国立理工学院);Haverford College(哈弗福德学院);Wollo University(沃尔洛大学);Pennsylvania State University(宾夕法尼亚州立大学);University of Hamburg(汉堡大学)
Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages
Georgia Institute of Technology(佐治亚理工学院);Samsung R&D Institute Philippines(三星菲律宾研发院);Sungkyunkwan University(成均馆大学);Tohoku University(东北大学);National University of Singapore(新加坡国立大学);University of Copenhagen(哥本哈根大学);University of Michigan(密歇根大学);Indian Institute of Technology Delhi(印度理工学院德里);Institute of Science Tokyo(东京科学研究院)
Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models
University of British Columbia(不列颠哥伦比亚大学);Zuse School(Zuse学校);Qatar Computing Research Institute (QCRI)(卡塔尔计算研究所);Hamad Bin Khalifa University(哈马德·本·哈利法大学)
Evaluating Text Style Transfer: A Nine-Language Benchmark for Text Detoxification
AIRI;Universidade de Santiago de Compostela(圣塞巴斯蒂安大学);Technical University of Munich (TUM)(慕尼黑技术大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Skoltech
MedAraBench: Large-Scale Arabic Medical Question Answering Dataset and Benchmark
Engineering Division, New York University Abu Dhabi(纽约大学阿布扎克分校工程系);Cleveland Clinic Abu Dhabi(阿布扎克克利夫兰诊所);Science Division, New York University Abu Dhabi(纽约大学阿布扎克分校科学系)
ComplexityMT: Benchmarking the Interaction Between Text Complexity and Machine Translation
University of Bath(巴斯大学);Cardiff University(卡迪夫大学);National University Philippines(菲律宾国家大学);MBZUAI(穆扎布伊人工智能研究所);University of Exeter(埃克塞特大学);INESC-ID Lisboa(里斯本INESC-ID);Instituto Universitário de Lisboa (ISCTE-IUL), ISTAR(里斯本大学研究所(ISCTE-IUL),ISTAR);National Research Council, Canada(加拿大国家研究委员会);King Abdulaziz University(阿卜杜勒-阿齐兹大学);Saudi Electronic University(沙特电子大学)