“我知道该看哪里”，但大语言模型知道吗？——绘制临床专家需求与非结构化数据抽取工具之间的鸿沟
"I Know Where to Look," But Does the LLM? Charting the Gaps Between Clinical Expert Needs and Unstructured Data Abstraction Tools
浏览论文内容
中文总结 AI 辅助
本研究通过与癌症研究团队共同设计LLM抽象系统Libretto，发现临床专家在多数任务中难以用LLM复现其标注直觉，揭示了AI数据工具与临床需求间的鸿沟及HCI研究挑战。
中文摘要 AI 辅助
临床数据抽象，即从患者记录中提炼结构化信息的过程，在推进癌症等疾病的知识方面发挥着关键作用。利用大型语言模型（LLM）进行信息抽取（IE）可以加速这一过程，但目前尚不清楚现有框架能否有效支持缺乏人工智能专业知识的临床研究人员。为解决这一问题，我们与七个癌症研究团队共同设计了一个名为Libretto的交互式基于LLM的抽象系统，然后评估了该系统帮助他们回答真实世界研究问题的能力。我们发现，虽然临床医生知道在患者笔记中何处以及如何标注复杂概念，但在十四项任务中的十二项中，他们在用LLM复现这些直觉时遇到了障碍。对笔记可靠性的情境化判断、难以引导“氛围编码”提示词以及僵化的评估策略，迫使信息抽取工作流程发生根本性改变。我们的结果凸显了人机交互（HCI）研究在弥合人工智能数据工作工具与临床用户需求之间差距方面所面临的开放性问题。
英文摘要
Clinical data abstraction, the process of distilling structured information from patient records, plays a key role in advancing knowledge about diseases such as cancer. Information extraction (IE) with large language models (LLMs) could accelerate this process, but it is unclear whether current frameworks effectively support clinical researchers without AI expertise. To address this, we co-designed an interactive LLM-based abstraction system called Libretto with seven cancer research teams, then evaluated the system's ability to help them answer real-world research questions. We found that while clinicians knew where and how to annotate complex concepts in patient notes, in twelve of fourteen tasks they faced barriers to replicating those intuitions with LLMs. Contextual note reliability judgments, difficulties in steering vibe-coded prompts, and inflexible evaluation strategies necessitated fundamental changes to the IE workflow. Our results highlight open problems for HCI research to bridge the gaps between AI data work tools and clinical users' needs.
发表机构
- University of California, San Francisco（旧金山加利福尼亚大学）
- Stanford University（斯坦福大学）
机构由 AI 辅助整理，请以论文原文为准。
