生成式大型语言模型与同类患者解答普通患者实验室检查结果的答案质量对比：评估研究
Quality of Answers of Generative Large Language Models vs Peer Patients for Interpreting Lab Test Results for Lay Patients: Evaluation Study
AI总结：
本研究评估了GPT-4等四个大型语言模型解答患者实验室检查问题的质量，发现GPT-4在相关性、正确性、有用性和安全性上优于其他模型及人类回答，但仍存在个性化不足等问题，并提出了改进方法。
摘要：
实验室检查结果通常令人困惑且难以理解。诸如ChatGPT之类的大型语言模型（LLMs）为患者获取问题解答开辟了一条充满希望的途径。我们旨在评估使用LLMs针对患者提出的实验室检查相关问题生成相关、准确、有用且无害回复的可行性，并识别可通过增强方法缓解的潜在问题。我们首先从Yahoo! Answers收集了与实验室检查结果相关的问答数据，并为本研究选取了53个问答对。使用LangChain框架和ChatGPT网络门户，我们生成了四个LLM（包括GPT-4、Meta LLaMA 2、MedAlpaca和ORCA_mini）对这53个问题的回复。我们首先使用基于标准问答相似度的评估指标（包括ROUGE、BLEU、METEOR、BERTScore）评估了它们答案的相似度。我们还利用基于LLM的评估器来判断目标模型在相关性、正确性、有用性和安全性方面是否优于基线模型。最后，我们与医学专家一起对选定的七个问题的所有回复进行了人工评估，评估方面与上述四个方面相同。Win Rate和医学专家评估的结果均表明，GPT-4的回复在所有四个方面（相关性、正确性、有用性和安全性）的得分均优于所有其他LLM回复和人类回复。然而，LLM的回复偶尔也会出现缺乏结合个人医疗背景的解读、陈述错误以及缺乏参考引用的问题。我们发现，与其他三个LLM和问答网站的人类回答相比，GPT-4的回复更准确、更有用、更相关且更安全。但是，也存在GPT-4回复不准确和缺乏个性化的情况。我们确定了多种提升LLM回复质量的方法。
英文摘要：
Lab results are often confusing and hard to understand. Large language models (LLMs) such as ChatGPT have opened a promising avenue for patients to get their questions answered. We aim to assess the feasibility of using LLMs to generate relevant, accurate, helpful, and unharmful responses to lab test-related questions asked by patients and to identify potential issues that can be mitigated with augmentation approaches. We first collected lab test results related question and answer data from Yahoo! Answers and selected 53 QA pairs for this study. Using the LangChain framework and ChatGPT web portal, we generated responses to the 53 questions from four LLMs including GPT-4, Meta LLaMA 2, MedAlpaca, and ORCA_mini. We first assessed the similarity of their answers using standard QA similarity-based evaluation metrics including ROUGE, BLEU, METEOR, BERTScore. We also utilized an LLM-based evaluator to judge whether a target model has higher quality in terms of relevance, correctness, helpfulness, and safety than the baseline model. Finally, we performed a manual evaluation with medical experts for all the responses to seven selected questions on the same four aspects. The results of Win Rate and medical expert evaluation both showed that GPT-4's responses achieved better scores than all the other LLM responses and human responses on all four aspects (relevance, correctness, helpfulness, and safety). However, LLM responses occasionally also suffer from a lack of interpretation in one's medical context, incorrect statements, and lack of references. We find that compared to other three LLMs and human answer from the Q&A website, GPT-4's responses are more accurate, helpful, relevant, and safer. However, there are cases which GPT-4 responses are inaccurate and not individualized. We identified a number of ways to improve the quality of LLM responses.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
