MedDoc-Bot：儿科高血压指南语境下大语言模型比较分析的聊天工具
MedDoc-Bot: A Chat Tool for Comparative Analysis of Large Language Models in the Context of the Pediatric Hypertension Guideline
AI总结：
本研究评估了Meditron等4个开源LLM解读PDF医疗指南的效能，以ESC儿童高血压指南为测试场景，开发MedDoc-Bot工具，结合专家评分与METEOR等指标发现Mistral和Llama-2表现较好，为医疗文档解读LLM发展提供参考。
摘要：
本研究聚焦于评估非商业开源大语言模型（LLMs）Meditron、MedAlpaca、Mistral和Llama-2在解读PDF格式医疗指南时的效能。作为特定测试场景，我们将这些模型应用于欧洲心脏病学会（ESC）提供的儿童和青少年高血压指南。利用Python库Streamlit，我们开发了一款用户友好的医疗文档聊天机器人工具（MedDoc-Bot）。该工具允许授权用户上传PDF文件并提出问题，由四个本地存储的LLM生成解释性响应。一名儿科专家通过制定从ESC指南中提取的问题和响应，提供了评估基准。专家根据模型生成响应的准确性和相关性进行评分。此外，我们评估了METEOR和chrF指标得分，以评估模型响应与参考答案的相似性。我们的研究发现，Llama-2和Mistral在指标评估中表现良好，但Llama-2在处理文本和表格数据时速度较慢。在人类评估中，我们观察到Mistral、Meditron和Llama-2生成的响应具有合理的准确性和相关性。本研究为LLM在医疗文档解读未来发展中的优势和局限性提供了宝贵见解。开源代码：https://github.com/yaseen28/MedDoc-Bot
英文摘要：
This research focuses on evaluating the non-commercial open-source large language models (LLMs) Meditron, MedAlpaca, Mistral, and Llama-2 for their efficacy in interpreting medical guidelines saved in PDF format. As a specific test scenario, we applied these models to the guidelines for hypertension in children and adolescents provided by the European Society of Cardiology (ESC). Leveraging Streamlit, a Python library, we developed a user-friendly medical document chatbot tool (MedDoc-Bot). This tool enables authorized users to upload PDF files and pose questions, generating interpretive responses from four locally stored LLMs. A pediatric expert provides a benchmark for evaluation by formulating questions and responses extracted from the ESC guidelines. The expert rates the model-generated responses based on their fidelity and relevance. Additionally, we evaluated the METEOR and chrF metric scores to assess the similarity of model responses to reference answers. Our study found that Llama-2 and Mistral performed well in metrics evaluation. However, Llama-2 was slower when dealing with text and tabular data. In our human evaluation, we observed that responses created by Mistral, Meditron, and Llama-2 exhibited reasonable fidelity and relevance. This study provides valuable insights into the strengths and limitations of LLMs for future developments in medical document interpretation. Open-Source Code: https://github.com/yaseen28/MedDoc-Bot
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
