arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.09053cs.HC
收藏

谁在何处失败?LLM和人类在内膜异位症超声报告提取中的错误模式

Who Fails Where? LLM and Human Error Patterns in Endometriosis Ultrasound Report Extraction

Univ. of Adelaide(阿德莱德大学);Robinson Inst., Univ. of Adelaide(阿德莱德大学罗宾逊研究所);McMaster University(麦克马斯特大学)

Haiyi Li, Yutong Li, Yiheng Chi, Alison Deslandes, Mathew Leonardi, Shay Freger, Yuan Zhang, Jodie Avery, M. Louise Hull, Hsiang-Ting Chen

AI总结:

本研究比较了LLM与人类在内膜异位症超声报告提取中的表现,发现LLM在语法一致性上优于人类,而人类在语义解释上更优,支持人机协作的工作流程。

摘要:

在本研究中,我们评估了一个本地部署的大型语言模型(LLM),将其转换为无结构的内膜异位症经阴道超声(eTVUS)扫描报告为结构化数据,用于影像信息学工作流程。在49份eTVUS报告中,我们比较了三个LLM(7B/8B和一个20B参数模型)与专家人类提取。20B模型实现了86.02%的平均准确率,显著优于较小的模型,证实了在处理复杂临床文本时规模的重要性。关键的是,我们识别出一个高度互补的错误模式:LLM在语法一致性(例如日期/数字格式)方面表现优异,而人类则容易出错;而人类专家在语义和上下文解释方面提供更优表现。我们还发现,LLM的语义错误是根本性的限制,无法通过简单的提示工程来缓解。这些发现强烈支持了人机协作(HITL)的工作流程,其中本地的LLM作为协作工具,而非完全替代。它自动化了常规的结构化工作,标记潜在的人为错误,使影像专家能够专注于高级语义验证。我们讨论了结构化报告和临床实践中交互式AI系统的影响。

英文摘要:

In this study, we evaluate a locally-deployed large-language model (LLM) to convert unstructured endometriosis transvaginal ultrasound (eTVUS) scan reports into structured data for imaging informatics workflows. Across 49 eTVUS reports, we compared three LLMs (7B/8B and a 20B-parameter model) against expert human extraction. The 20B model achieved a mean accuracy of 86.02%, substantially outperforming smaller models and confirming the importance of scale in handling complex clinical text. Crucially, we identified a highly complementary error profile: the LLM excelled at syntactic consistency (e.g., date/numeric formatting) where humans faltered, while human experts provided superior semantic and contextual interpretation. We also found that the LLM's semantic errors were fundamental limitations that could not be mitigated by simple prompt engineering. These findings strongly support a human-in-the-loop (HITL) workflow in which the on-premise LLM serves as a collaborative tool, not a full replacement. It automates routine structuring and flags potential human errors, enabling imaging specialists to focus on high-level semantic validation. We discuss implications for structured reporting and interactive AI systems in clinical practice.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑