可解释大语言模型在泌尿外科委员会考题中实现超人表现并整合欧洲泌尿外科学会指南上下文：UroBot研究
Superhuman performance in urology board questions by an explainable large language model enabled for context integration of the European Association of Urology guidelines: the UroBot study
AI总结：
本研究构建泌尿外科专用聊天机器人UroBot，结合RAG与EAU指南，在EBU考题上平均正确率达88.4%，超越现有模型和泌尿外科医生，且具备临床可验证性。
摘要：
大语言模型（LLMs）通过广泛使用医学文献正在革新医学问答（medQA）。然而，其性能常受限于过时的训练数据和缺乏可解释性，这限制了临床应用。本研究旨在创建并评估泌尿外科专用聊天机器人UroBot，将其与最先进模型及泌尿外科医生在泌尿外科委员会考题上的表现进行比较，确保完全临床可验证性。UroBot基于OpenAI的GPT-3.5、GPT-4和GPT-4o模型开发，采用检索增强生成（RAG）及欧洲泌尿外科学会（EAU）2023年最新指南。评估包括对200道欧洲泌尿外科委员会（EBU）在职评估（ISA）考题进行十次运行，性能通过平均正确回答率（RoCA）衡量。UroBot-4o的平均RoCA达到88.4%，超过GPT-4o的77.6%达10.8个百分点。它同时具备临床可验证性，并展现出最高的运行一致性，如Fleiss' Kappa（k = 0.979）所示。相比之下，文献报道的泌尿外科医生在委员会考题上的平均表现为68.7%。UroBot的临床可验证性及在委员会考题上相较于现有模型和泌尿外科医生的优越准确性，凸显了其临床整合潜力。研究还提供了进一步开发UroBot所需的代码和说明。
英文摘要：
Large Language Models (LLMs) are revolutionizing medical Question-Answering (medQA) through extensive use of medical literature. However, their performance is often hampered by outdated training data and a lack of explainability, which limits clinical applicability. This study aimed to create and assess UroBot, a urology-specialized chatbot, by comparing it with state-of-the-art models and the performance of urologists on urological board questions, ensuring full clinician-verifiability. UroBot was developed using OpenAI's GPT-3.5, GPT-4, and GPT-4o models, employing retrieval-augmented generation (RAG) and the latest 2023 guidelines from the European Association of Urology (EAU). The evaluation included ten runs of 200 European Board of Urology (EBU) In-Service Assessment (ISA) questions, with performance assessed by the mean Rate of Correct Answers (RoCA). UroBot-4o achieved an average RoCA of 88.4%, surpassing GPT-4o by 10.8%, with a score of 77.6%. It was also clinician-verifiable and exhibited the highest run agreement as indicated by Fleiss' Kappa (k = 0.979). By comparison, the average performance of urologists on board questions, as reported in the literature, is 68.7%. UroBot's clinician-verifiable nature and superior accuracy compared to both existing models and urologists on board questions highlight its potential for clinical integration. The study also provides the necessary code and instructions for further development of UroBot.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
