arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.14567cs.CL
收藏

WangLab at MEDIQA-M3G 2024: 使用大型语言模型的多模态医学答案生成

WangLab at MEDIQA-M3G 2024: Multimodal Medical Answer Generation using Large Language Models

University of Toronto(多伦多大学);Vector Institute(向量研究所);Princess Margaret Cancer Centre, University Health Network(大学健康网络玛格丽特公主癌症中心);Lunenfeld-Tanenbaum Research Institute, Sinai Health System(西奈健康系统勒嫩菲尔德-塔南鲍姆研究所);Peter Munk Cardiac Centre, University Health Network(大学健康网络彼得·芒克心脏中心);AI Hub, University Health Network(大学健康网络人工智能中心)

Ronald Xie, Steven Palayew, Augustin Toma, Gary Bader, Bo Wang

AI总结:

本文介绍了WangLab参加MEDIQA-M3G 2024共享任务的两个方案,分别基于Claude 3 Opus API调用和CLIP风格图像分类,在英语类别中取得第一和第二名,并指出这两种方法在医学视觉问答中具有进一步研究潜力。

摘要:

本文概述了我们提交给MEDIQA2024多语言和多模态医学答案生成(M3G)共享任务的方案。我们报告了该任务英语类别下两个独立解决方案的结果:第一个方案涉及对Claude 3 Opus API进行两次连续的API调用;第二个方案涉及以CLIP的风格训练图像-疾病标签联合嵌入,用于图像分类。这两个解决方案在竞赛排行榜上分别获得第1名和第2名,大幅优于次优方案。此外,我们讨论了赛后实验中获得的见解。尽管由于共享任务的难度以及医学视觉问答普遍具有的挑战性,这两个解决方案的性能仍有很大的提升空间,但我们认为多阶段LLM方法和CLIP图像分类方法是值得进一步研究的途径。

英文摘要:

This paper outlines our submission to the MEDIQA2024 Multilingual and Multimodal Medical Answer Generation (M3G) shared task. We report results for two standalone solutions under the English category of the task, the first involving two consecutive API calls to the Claude 3 Opus API and the second involving training an image-disease label joint embedding in the style of CLIP for image classification. These two solutions scored 1st and 2nd place respectively on the competition leaderboard, substantially outperforming the next best solution. Additionally, we discuss insights gained from post-competition experiments. While the performance of these two solutions have significant room for improvement due to the difficulty of the shared task and the challenging nature of medical visual question answering in general, we identify the multi-stage LLM approach and the CLIP image classification approach as promising avenues for further investigation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑