EPPCMinerBen：一种用于通过患者门户评估大语言模型在电子患者-提供者沟通中的新基准
EPPCMinerBen: A Novel Benchmark for Evaluating Large Language Models on Electronic Patient-Provider Communication via the Patient Portal
AI总结：
EPPCMinerBen是一个用于评估大语言模型在电子患者-提供者沟通中表现的新基准，通过患者门户数据验证了大型模型在证据提取和分类任务中的优势。
摘要：
医疗领域中的有效沟通对于治疗结果和依从性至关重要。随着患者-提供者交流转向安全信息传递，分析电子患者沟通（EPPC）数据既是必要的也是具有挑战性的。我们介绍了EPPCMinerBen，这是一个用于评估LLM在检测沟通模式和从电子患者-提供者信息中提取见解方面的基准。EPPCMinerBen包含三个子任务：代码分类、子代码分类和证据提取。使用来自耶鲁纽黑文医院患者门户的752条安全信息中的1933条专家注释句子，它评估LLM在识别沟通意图和支持文本方面的表现。基准涵盖了各种LLM在零样本和少样本设置下的表现，数据将通过国家癌症登记处数据服务发布。模型性能在不同任务和设置中有所差异。Llama-3.1-70B在证据提取（F1: 82.84%）中领先，并在分类中表现良好。Llama-3.3-70b-Instruct在代码分类（F1: 67.03%）中优于所有模型。DeepSeek-R1-Distill-Qwen-32B在子代码分类（F1: 48.25%）中表现出色，而sdoh-llama-3-70B表现稳定。较小的模型在子代码分类中表现欠佳（F1>30%）。少样本提示提高了大多数任务的表现。我们的结果表明，大型、指令微调模型在EPPCMinerBen任务中表现更好，特别是在证据提取方面，而较小的模型在细粒度推理上挣扎。EPPCMinerBen为话语层面的理解提供了基准，支持未来关于模型泛化和患者-提供者沟通分析的工作。关键词：电子患者-提供者沟通，大语言模型，数据收集，提示工程
英文摘要：
Effective communication in health care is critical for treatment outcomes and adherence. With patient-provider exchanges shifting to secure messaging, analyzing electronic patient-communication (EPPC) data is both essential and challenging. We introduce EPPCMinerBen, a benchmark for evaluating LLMs in detecting communication patterns and extracting insights from electronic patient-provider messages. EPPCMinerBen includes three sub-tasks: Code Classification, Subcode Classification, and Evidence Extraction. Using 1,933 expert annotated sentences from 752 secure messages of the patient portal at Yale New Haven Hospital, it evaluates LLMs on identifying communicative intent and supportive text. Benchmarks span various LLMs under zero-shot and few-shot settings, with data to be released via the NCI Cancer Data Service. Model performance varied across tasks and settings. Llama-3.1-70B led in evidence extraction (F1: 82.84%) and performed well in classification. Llama-3.3-70b-Instruct outperformed all models in code classification (F1: 67.03%). DeepSeek-R1-Distill-Qwen-32B excelled in subcode classification (F1: 48.25%), while sdoh-llama-3-70B showed consistent performance. Smaller models underperformed, especially in subcode classification (>30% F1). Few-shot prompting improved most tasks. Our results show that large, instruction-tuned models generally perform better in EPPCMinerBen tasks, particularly evidence extraction while smaller models struggle with fine-grained reasoning. EPPCMinerBen provides a benchmark for discourse-level understanding, supporting future work on model generalization and patient-provider communication analysis. Keywords: Electronic Patient-Provider Communication, Large language models, Data collection, Prompt engineering
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
