HealthQ：揭示医疗对话中LLM链的提问能力
HealthQ: Unveiling Questioning Capabilities of LLM Chains in Healthcare Conversations
AI总结：
HealthQ提出首个系统评估医疗对话中LLM链提问能力的框架，通过多种链和评判器验证，证明高质量问题能提升患者信息收集效果。
摘要：
在数字医疗中，有效的患者护理需要大型语言模型（LLM）不仅能够回答问题，还能通过精心设计的询问主动收集关键信息。本文介绍了HealthQ，一个用于评估LLM医疗链提问能力的新框架。通过实现先进的LLM链，包括检索增强生成（RAG）、思维链（CoT）和反思链，HealthQ评估了这些链在引出全面且相关的患者信息方面的有效性。为此，我们整合了一个LLM评判器，用于评估生成的问题在特异性、相关性和有用性等指标上的表现，同时将这些评估与传统的自然语言处理（NLP）指标（如ROUGE和基于命名实体识别（NER）的集合比较）对齐。我们使用从公共医疗数据集ChatDoctor和MTS-Dialog构建的两个自定义数据集验证了HealthQ，并展示了其在多个LLM评判模型（包括GPT-3.5、GPT-4和Claude）上的稳健性。我们的贡献有三方面：我们提出了首个用于评估医疗对话中提问能力的系统框架，建立了一种模型无关的评估方法，并提供了高质量问题与改善患者信息引出之间关联的实证证据。
英文摘要：
Effective patient care in digital healthcare requires large language models (LLMs) that not only answer questions but also actively gather critical information through well-crafted inquiries. This paper introduces HealthQ, a novel framework for evaluating the questioning capabilities of LLM healthcare chains. By implementing advanced LLM chains, including Retrieval-Augmented Generation (RAG), Chain of Thought (CoT), and reflective chains, HealthQ assesses how effectively these chains elicit comprehensive and relevant patient information. To achieve this, we integrate an LLM judge to evaluate generated questions across metrics such as specificity, relevance, and usefulness, while aligning these evaluations with traditional Natural Language Processing (NLP) metrics like ROUGE and Named Entity Recognition (NER)-based set comparisons. We validate HealthQ using two custom datasets constructed from public medical datasets, ChatDoctor and MTS-Dialog, and demonstrate its robustness across multiple LLM judge models, including GPT-3.5, GPT-4, and Claude. Our contributions are threefold: we present the first systematic framework for assessing questioning capabilities in healthcare conversations, establish a model-agnostic evaluation methodology, and provide empirical evidence linking high-quality questions to improved patient information elicitation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
