语言模型与检索增强生成在诊断报告自动化结构化数据提取中的应用
Language Models and Retrieval Augmented Generation for Automated Structured Data Extraction from Diagnostic Reports
AI总结：
本研究利用开放权重语言模型和检索增强生成，自动化提取放射学与病理学报告中的结构化临床信息，最佳模型准确率超98%和90%，验证了本地隐私保护下临床数据提取的可行性。
摘要：
目的：开发并评估一种自动化系统，利用开放权重大型语言模型（LMs）和检索增强生成（RAG）从非结构化的放射学和病理学报告中提取结构化临床信息，并评估模型配置变量对提取性能的影响。方法与材料：研究使用了两个数据集：7,294份标注了脑肿瘤报告和数据系统（BT-RADS）评分的放射学报告，以及2,154份标注了异柠檬酸脱氢酶（IDH）突变状态的病理学报告。开发了一个自动化流水线，用于基准测试各种LM和RAG配置的性能。系统评估了模型大小、量化、提示策略、输出格式和推理参数的影响。结果：性能最佳的模型在从放射学报告中提取BT-RADS评分方面达到了超过98%的准确率，在从病理学报告中提取IDH突变状态方面达到了超过90%的准确率。表现最好的模型是医学微调的llama3。更大、更新以及领域微调的模型始终优于较旧和较小的模型。模型量化对性能的影响最小。少样本提示显著提高了准确性。RAG改善了复杂病理学报告的性能，但对较短的放射学报告没有改善。结论：开放LM在从非结构化临床报告中自动化提取结构化临床数据方面展现出巨大潜力，并支持本地隐私保护应用。仔细的模型选择、提示工程以及使用标注数据进行半自动化优化对于实现最佳性能至关重要。这些方法在研究工作流程中可能足够可靠，突显了在医疗数据提取中人机协作的潜力。
英文摘要：
Purpose: To develop and evaluate an automated system for extracting structured clinical information from unstructured radiology and pathology reports using open-weights large language models (LMs) and retrieval augmented generation (RAG), and to assess the effects of model configuration variables on extraction performance. Methods and Materials: The study utilized two datasets: 7,294 radiology reports annotated for Brain Tumor Reporting and Data System (BT-RADS) scores and 2,154 pathology reports annotated for isocitrate dehydrogenase (IDH) mutation status. An automated pipeline was developed to benchmark the performance of various LMs and RAG configurations. The impact of model size, quantization, prompting strategies, output formatting, and inference parameters was systematically evaluated. Results: The best performing models achieved over 98% accuracy in extracting BT-RADS scores from radiology reports and over 90% for IDH mutation status extraction from pathology reports. The top model being medical fine-tuned llama3. Larger, newer, and domain fine-tuned models consistently outperformed older and smaller models. Model quantization had minimal impact on performance. Few-shot prompting significantly improved accuracy. RAG improved performance for complex pathology reports but not for shorter radiology reports. Conclusions: Open LMs demonstrate significant potential for automated extraction of structured clinical data from unstructured clinical reports with local privacy-preserving application. Careful model selection, prompt engineering, and semi-automated optimization using annotated data are critical for optimal performance. These approaches could be reliable enough for practical use in research workflows, highlighting the potential for human-machine collaboration in healthcare data extraction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
