PerMedCQA：波斯语医学消费者问答中大型语言模型的基准评测
PerMedCQA: Benchmarking Large Language Models on Medical Consumer Question Answering in Persian Language
AI总结：
本文提出PerMedCQA，首个波斯语医学消费者问答基准，含68,138个问答对，并利用MedJudge框架评估多种LLM，揭示多语言医学问答挑战，推动更准确的医学辅助系统开发。
摘要：
医学消费者问答（CQA）对于通过提供个性化和可靠的健康信息来增强患者能力至关重要。尽管近期大型语言模型（LLM）在医学问答方面取得了进展，但面向消费者和多语言的资源，尤其是在波斯语等低资源语言中，仍然稀缺。为填补这一空白，我们提出了PerMedCQA，这是首个用于评估LLM在真实世界消费者生成的医学问题上表现的波斯语基准。PerMedCQA从大型医学问答论坛中精选，包含68,138个问答对，经过从初始87,780条原始条目中仔细的数据清洗而得。我们评估了多种最先进的多语言和指令微调LLM，利用MedJudge（一种由LLM评分器驱动的新型基于评分标准的评估框架），并与专家人工标注者进行验证。我们的结果突出了多语言医学问答中的关键挑战，并为开发更准确和上下文感知的医学辅助系统提供了宝贵见解。数据公开于https://huggingface.co/datasets/NaghmehAI/PerMedCQA。
英文摘要：
Medical consumer question answering (CQA) is crucial for empowering patients by providing personalized and reliable health information. Despite recent advances in large language models (LLMs) for medical QA, consumer-oriented and multilingual resources, particularly in low-resource languages like Persian, remain sparse. To bridge this gap, we present PerMedCQA, the first Persian-language benchmark for evaluating LLMs on real-world, consumer-generated medical questions. Curated from a large medical QA forum, PerMedCQA contains 68,138 question-answer pairs, refined through careful data cleaning from an initial set of 87,780 raw entries. We evaluate several state-of-the-art multilingual and instruction-tuned LLMs, utilizing MedJudge, a novel rubric-based evaluation framework driven by an LLM grader, validated against expert human annotators. Our results highlight key challenges in multilingual medical QA and provide valuable insights for developing more accurate and context-aware medical assistance systems. The data is publicly available on https://huggingface.co/datasets/NaghmehAI/PerMedCQA
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
