mamabench 和 mamaretrieval：评估孕产妇、新生儿和生殖健康领域医学检索增强生成的基准
mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health
AI总结：
针对助产士咨询的孕产妇、新生儿和生殖健康问题，构建了包含25,949个问题的QA基准mamabench和基于3,185个查询的段落级相关性基准mamaretrieval，采用分级相关标注和标签质量审计。
摘要：
医学问答基准很少涵盖助产士提出的孕产妇、新生儿、儿童和生殖健康问题，而且据我们所知，目前没有公开的用于孕产妇健康指南检索的段落级相关性基准。我们发布了两个填补这些空白的基准。mamabench 是一个范围过滤的问答集，包含从七个现有专家来源整理的25,949个项目，涵盖选择题、简答题和评分标准题；为了帮助用户校准评分标准题中使用的LLM评判器，我们将HealthBench的医生标注元评估重新限定到该领域。mamaretrieval 将3,185个临床查询与在63,650个段落的孕产妇健康指南语料库上的分级（0-6）相关性标签配对，使用一个分解的评分标准来区分回答查询的段落和仅仅涉及该主题的段落。三个决策塑造了这两个基准：整理和过滤专家来源而非编写问题；对相关性进行分级而非二值化；测量并披露标签的局限性——范围分类器一致性、前沿评判器检查和池化完整性审计——而不是将其视为标准答案。一篇配套论文使用这些基准评估了一个部署在设备上的助手；两者均公开用于研究。
英文摘要：
Medical question-answering benchmarks rarely cover the maternal, neonatal, child, and reproductive-health questions a nurse-midwife asks, and, to our knowledge, no public chunk-level relevance benchmark exists for maternal-health guideline retrieval. We release two benchmarks that fill these gaps. mamabench is a scope-filtered QA set of 25,949 items assembled from seven existing expert-authored sources across multiple-choice, short-answer, and rubric-graded tracks; to help users calibrate the LLM judge that scores the rubric track, we re-scope HealthBench's physician-labelled meta-evaluation to the domain. mamaretrieval pairs 3,185 clinical queries with graded (0-6) relevance labels over a 63,650-chunk maternal-health guideline corpus, using a decomposed rubric that distinguishes a chunk that answers a query from one merely on its topic. Three decisions shape both: assemble and filter expert sources rather than author questions, grade relevance rather than binarise it, and measure and disclose the limits of the labels -- scope-classifier agreement, a frontier-judge check, and a pooling-completeness audit -- rather than treat them as an oracle. A companion paper uses the benchmarks to evaluate a deployed on-device assistant; both are released openly for research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
