OrthoDoc：用于计算机断层扫描辅助诊断的多模态大语言模型
OrthoDoc: Multimodal Large Language Model for Assisting Diagnosis in Computed Tomography
AI总结：
针对通用大语言模型难精准处理CT复杂纹理、诊断辅助效果受限的问题，研究提出基于12万CT数据训练、带医学知识RAG模块的OrthoDoc多模态大模型，在骨科疾病诊断上优于GPT-4等商用模型，且罕见病例泛化性强。
摘要：
多模态大语言模型（MLLMs）已在通用图像处理领域取得显著成效，其新兴的任务泛化能力与自由对话能力可极大助力医疗诊断辅助，帮助患者更好地了解自身病情，增进医患信任。计算机断层扫描（CT）是一种用于捕捉患者病情内部机制的无创成像技术，应用十分广泛。但在过往研究中，这类成像数据的复杂纹理特征使得算法难以实现精准判读，制约了通用大语言模型在诊断辅助中的表现。为解决这一问题，我们开发了面向CT诊断的多模态大语言模型OrthoDoc。OrthoDoc基于12万张CT图像及诊断报告训练，内置检索增强生成（RAG，Retrieval-Augmented Generation）模块，该模块依托海量医学文献、教科书及解释性数据，可有效缓解模型幻觉问题。因此，OrthoDoc不仅能够处理复杂的CT图像，还可实现对医学知识与语言的存储、理解和推理。在大量实验中，OrthoDoc的表现优于以GPT-4为代表的商用模型，展现出更出色的诊断能力与准确率。具体而言，OrthoDoc在骨折、关节炎、肿瘤等常见骨科疾病的诊断上显著超越现有模型；此外，在处理罕见复杂病例时，OrthoDoc也表现出较强的泛化性与稳定性。
英文摘要：
Multimodal large language models (MLLMs) have achieved significant success in the general field of image processing. Their emerging task generalization and freeform conversational capabilities can greatly facilitate medical diagnostic assistance, helping patients better understand their conditions and enhancing doctor-patient trust. Computed Tomography (CT) is a non-invasive imaging technique used to capture the internal mechanisms of a patient's condition and is widely utilized. However, in past research, the complex textural features of this imaging data have made accurate interpretation by algorithms challenging, impeding the performance of general LLMs in diagnostic assistance. To address this, we developed OrthoDoc, a MLLM designed for CT diagnostics. OrthoDoc is trained on 120,000 CT images and diagnostic reports and includes a Retrieval-Augmented Generation (RAG) module capable of effectively mitigating model hallucinations. This module is informed by extensive medical literature, textbooks, and explanatory data. Thus, OrthoDoc not only processes complex CT images but also stores, understands, and reasons over medical knowledge and language. In extensive experiments, OrthoDoc outperforms commercial models led by GPT-4, demonstrating superior diagnostic capabilities and accuracy. Specifically, OrthoDoc significantly surpasses existing models in the diagnosis of common orthopedic conditions such as fractures, arthritis, and tumors. Additionally, OrthoDoc exhibits robust generalization and stability when handling rare and complex cases.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
