CliniQ：面向电子健康记录检索的多维度基准与语义匹配评估
CliniQ: A Multi-faceted Benchmark for Electronic Health Record Retrieval with Semantic Match Assessment
AI总结：
本文提出公开EHR检索基准CliniQ，涵盖单患者与多患者检索、语义匹配类型评估，并通过大规模实验揭示BM25及通用稠密检索器的竞争力，为EHR检索研究提供新基准。
摘要：
电子健康记录（EHR）检索在多种临床任务中发挥着关键作用，但其发展一直因缺乏公开可用的基准而受到严重制约。本文引入了一个新颖的公开EHR检索基准CliniQ，以填补这一空白。我们考虑了两种检索设置：单患者检索和多患者检索，以反映各种真实世界场景。单患者检索侧重于在单个患者病历中找到相关部分，而多患者检索则涉及从多个患者中检索EHR。我们基于MIMIC-III中的1,000份出院小结以及ICD编码和处方标签构建基准，并通过进一步利用强大的大语言模型（LLM）作为标注者，收集了1,246个唯一查询和77,206条相关性判断。此外，我们通过将匹配类型划分为字符串匹配和四种语义匹配，引入了一种对EHR检索中语义差距问题的新颖评估。在我们提出的基准上，我们对从传统精确匹配到流行的稠密检索器等多种检索方法进行了全面评估。实验发现，BM25建立了强大的基线，其性能与稠密检索器相当，且通用领域的稠密检索器出人意料地优于专为医学领域设计的检索器。对各种匹配类型的深入分析揭示了不同方法的优势与不足，为针对性改进提供了启示。我们相信，该基准将激励研究界推进EHR检索系统的发展。
英文摘要：
Electronic Health Record (EHR) retrieval plays a pivotal role in various clinical tasks, but its development has been severely impeded by the lack of publicly available benchmarks. In this paper, we introduce a novel public EHR retrieval benchmark, CliniQ, to address this gap. We consider two retrieval settings: Single-Patient Retrieval and Multi-Patient Retrieval, reflecting various real-world scenarios. Single-Patient Retrieval focuses on finding relevant parts within a patient note, while Multi-Patient Retrieval involves retrieving EHRs from multiple patients. We build our benchmark upon 1,000 discharge summary notes along with the ICD codes and prescription labels from MIMIC-III, and collect 1,246 unique queries with 77,206 relevance judgments by further leveraging powerful LLMs as annotators. Additionally, we include a novel assessment of the semantic gap issue in EHR retrieval by categorizing matching types into string match and four types of semantic matches. On our proposed benchmark, we conduct a comprehensive evaluation of various retrieval methods, ranging from conventional exact match to popular dense retrievers. Our experiments find that BM25 sets a strong baseline and performs competitively to the dense retrievers, and general domain dense retrievers surprisingly outperform those designed for the medical domain. In-depth analyses on various matching types reveal the strengths and drawbacks of different methods, enlightening the potential for targeted improvement. We believe that our benchmark will stimulate the research communities to advance EHR retrieval systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
