当度量标准产生分歧：自动相似性与LLM作为评判者在临床对话评估中的对比
When Metrics Disagree: Automatic Similarity vs. LLM-as-a-Judge for Clinical Dialogue Evaluation
AI总结：
本文研究了在临床对话评估中，LLM作为评判者与传统自动相似性度量之间的分歧，通过LoRA技术对Llama-2-7B进行领域适应，并发现自动化指标可能无法完全反映临床实用性。
摘要：
随着大型语言模型（LLMs）在医疗领域中被越来越多地用于解决复杂问题，确保其可靠性成为关键挑战。最近的研究表明，通用LLMs在临床环境中经常表现不佳，有时会产生误导性指导。为减轻这些风险，本研究专注于使用低秩适应（LoRA）技术对Llama-2-7B进行领域特定适应。通过在Transformer层中注入可训练的低秩矩阵，我们利用真实的患者-医生对话文本高效地适应模型，同时保留基础模型的基础知识。我们的目标是通过捕捉临床对话的专业细微差别，提高对医疗查询的精度和上下文相关性。由于大规模人工验证资源消耗大，模型性能通过双轨框架评估：Track A使用传统词法相似性度量（如BLEU、ROUGE），而Track B则采用“LLM作为评判者”范式使用GPT-4进行语义评估。我们的结果表明，虽然LoRA增强的模型在所有定量词法维度上均取得显著提升，但在GPT-4评估中出现了深刻分歧，GPT-4略微偏向基线模型的对话流畅性。这种度量分歧突显了一个关键发现：传统自动化分数可能无法完全反映临床实用性。因此，我们提出，尽管自动化指标和LLM评判者作为有价值的开发代理，但严格的人类医学专家验证仍然是安全部署LLM在医疗环境中的不可或缺要求。
英文摘要：
As Large Language Models (LLMs) are increasingly integrated into healthcare to address complex inquiries, ensuring their reliability remains a critical challenge. Recent studies have highlighted that generic LLMs often struggle in clinical contexts, occasionally producing misleading guidance. To mitigate these risks, this research focuses on the domain-specific adaptation of \textbf{Llama-2-7B} using the \textbf{Low-Rank Adaptation (LoRA)} technique. By injecting trainable low-rank matrices into the Transformer layers, we efficiently adapted the model using authentic patient-physician transcripts while preserving the foundational knowledge of the base model. Our objective was to enhance precision and contextual relevance in responding to medical queries by capturing the specialized nuances of clinical discourse. Due to the resource-intensive nature of large-scale human validation, the model's performance was evaluated through a dual-track framework: \textbf{Track A} utilized traditional lexical similarity metrics (e.g., BLEU, ROUGE), while \textbf{Track B} employed an "LLM-as-a-Judge" paradigm using GPT-4 for semantic assessment. Our results demonstrate that while the LoRA-enhanced model achieved significant improvements across all quantitative lexical dimensions, a profound disagreement surfaced in the GPT-4 evaluation, which marginally favored the baseline model's conversational flow. This metric divergence underscores a pivotal finding: traditional automated scores may not fully reflect clinical utility. Consequently, we propose that while automated metrics and LLM judges serve as valuable developmental proxies, rigorous validation by human medical experts remains an indispensable requirement for the safe deployment of LLMs in healthcare settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
