MultiMedEval：用于评估医学视觉-语言模型的基准和工具包
MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models
AI总结：
MultiMedEval通过23个数据集和11个医学领域，评估六种多模态任务，提供开源工具简化VLM评估，促进公平统一的基准测试。
摘要：
我们介绍了MultiMedEval，一个开源工具包，用于公平且可重复地评估大型医学视觉-语言模型（VLM）。MultiMedEval全面评估模型在六个多模态任务上的性能，覆盖23个数据集和11个医学领域。所选任务和性能指标基于其在社区中的广泛应用和多样性，确保对模型整体泛化能力的全面评估。我们开源了一个Python工具包（github.com/corentin-ryr/MultiMedEval），提供简单的接口和设置过程，使任何VLM的评估只需几行代码即可完成。我们的目标是简化VLM评估的复杂 landscape，从而促进未来模型的公平和统一基准测试。
英文摘要：
We introduce MultiMedEval, an open-source toolkit for fair and reproducible evaluation of large, medical vision-language models (VLM). MultiMedEval comprehensively assesses the models' performance on a broad array of six multi-modal tasks, conducted over 23 datasets, and spanning over 11 medical domains. The chosen tasks and performance metrics are based on their widespread adoption in the community and their diversity, ensuring a thorough evaluation of the model's overall generalizability. We open-source a Python toolkit (github.com/corentin-ryr/MultiMedEval) with a simple interface and setup process, enabling the evaluation of any VLM in just a few lines of code. Our goal is to simplify the intricate landscape of VLM evaluation, thus promoting fair and uniform benchmarking of future models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
