基于知识的学习在Text-RAG和Image-RAG中的应用
Knowledge-based learning in Text-RAG and Image-RAG
AI总结：
本研究通过比较基于文本和图像的RAG方法，探讨了如何利用外部知识减少幻觉问题并提升胸部X光图像疾病检测的准确性。
摘要：
本研究分析并比较了基于Vision Transformer(EVA-ViT)的多模态图像编码器与LlaMA或ChatGPT LLM在减少幻觉问题和检测胸部X光图像疾病方面的多模态方法。在此研究中，我们利用NIH胸部X光图像训练模型，并在基于图像的RAG、基于文本的RAG和基线模型之间进行比较。结果表明，基于文本的RAG通过使用外部知识信息有效减少了幻觉问题，而基于图像的RAG通过使用KNN方法提高了预测信心和校准。此外，GPT LLM在幻觉率和预期校准误差(ECE)方面优于Llama LLM模型。本研究展示了数据不平衡的挑战，复杂多阶段结构，但提出了大规模经验环境和平衡示例的使用建议。
英文摘要：
This research analyzed and compared the multi-modal approach in the Vision Transformer(EVA-ViT) based image encoder with the LlaMA or ChatGPT LLM to reduce the hallucination problem and detect diseases in chest x-ray images. In this research, we utilized the NIH Chest X-ray image to train the model and compared it in image-based RAG, text-based RAG, and baseline. [3] [5] In a result, the text-based RAG[2] e!ectively reduces the hallucination problem by using external knowledge information, and the image-based RAG improved the prediction con"dence and calibration by using the KNN methods. [4] Moreover, the GPT LLM showed better performance, a low hallucination rate, and better Expected Calibration Error(ECE) than Llama Llama-based model. This research shows the challenge of data imbalance, a complex multi-stage structure, but suggests a large experience environment and a balanced example of use.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
