arXiv 2507.146152025-07-19cs.CLcs.AI

肯尼亚初级保健中用于情境化模型测试的检索增强临床基准：一篇方法学论文

Retrieval-Augmented Clinical Benchmarking for Contextual Model Testing in Kenyan Primary Care: A Methodology Paper

Qhala；Kenya Medical Association（肯尼亚医学协会）；Africa CDC（非洲疾病预防控制中心）；AfCFTA（非洲大陆自由贸易区）

Fred Mutisya, Shikoh Gitau, Christine Syovata, Diana Oigara, Ibrahim Matende, Muna Aden, Munira Ali, Ryan Nyotu, Diana Marion, Job Nyangena, Nasubo Ongoma, Keit… 展开作者

Fred Mutisya, Shikoh Gitau, Christine Syovata, Diana Oigara, Ibrahim Matende, Muna Aden, Munira Ali, Ryan Nyotu, Diana Marion, Job Nyangena, Nasubo Ongoma, Keith Mbae, Elizabeth Wamicha, Eric Mibuari, Jean Philbert Nsengemana, Talkmore Chidede

备注：: 29 pages, 6 figs, 6 tables. Companion methods paper forthcoming

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出一种基于检索增强生成和肯尼亚国家指南的临床基准构建方法，创建 Alama Health QA 数据集并引入推理、安全与适应性评估指标，揭示 LLM 在非洲初级保健场景中的性能差距。

摘要：

大型语言模型（LLMs）在改善资源匮乏地区的医疗可及性方面具有潜力，但它们在非洲初级保健中的有效性仍未得到充分探索。我们提出了一种方法，用于创建以肯尼亚二级和三级临床护理为重点的基准数据集和评估框架。我们的方法使用检索增强生成（RAG）将临床问题锚定在肯尼亚国家指南中，确保与当地标准保持一致。这些指南经过数字化、分块和索引，以用于语义检索。随后，向 Gemini Flash 2.0 Lite 提供指南摘录作为提示，以生成逼真的临床场景、多项选择题以及基于推理的英语和斯瓦希里语答案。肯尼亚医生共同创建并完善了该数据集，盲法专家评审过程确保了临床准确性、清晰性和文化适宜性。由此产生的 Alama Health QA 数据集包含数千个与监管要求一致的问答对，涵盖常见门诊疾病。除准确性外，我们还引入了评估临床推理、安全性和适应性的评估指标，例如罕见病例检测（Haystack 中的针）、逐步逻辑（决策点）和情境适应性。初步结果显示，当 LLMs 应用于本地化场景时，存在显著的性能差距，这与 LLM 在非洲医学内容上的准确性低于基于美国的基准这一发现一致。这项工作为以指南为驱动、动态化的基准测试提供了一个可复制的模型，以支持非洲卫生系统中安全的人工智能部署。

英文摘要：

Large Language Models(LLMs) hold promise for improving healthcare access in low-resource settings, but their effectiveness in African primary care remains underexplored. We present a methodology for creating a benchmark dataset and evaluation framework focused on Kenyan Level 2 and 3 clinical care. Our approach uses retrieval augmented generation (RAG) to ground clinical questions in Kenya's national guidelines, ensuring alignment with local standards. These guidelines were digitized, chunked, and indexed for semantic retrieval. Gemini Flash 2.0 Lite was then prompted with guideline excerpts to generate realistic clinical scenarios, multiple-choice questions, and rationale based answers in English and Swahili. Kenyan physicians co-created and refined the dataset, and a blinded expert review process ensured clinical accuracy, clarity, and cultural appropriateness. The resulting Alama Health QA dataset includes thousands of regulator-aligned question answer pairs across common outpatient conditions. Beyond accuracy, we introduce evaluation metrics that test clinical reasoning, safety, and adaptability such as rare case detection (Needle in the Haystack), stepwise logic (Decision Points), and contextual adaptability. Initial results reveal significant performance gaps when LLMs are applied to localized scenarios, consistent with findings that LLM accuracy is lower on African medical content than on US-based benchmarks. This work offers a replicable model for guideline-driven, dynamic benchmarking to support safe AI deployment in African health systems.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。