ACE-$M^3$：多模态医学模型的自动能力评估器
ACE-$M^3$: Automatic Capability Evaluator for Multimodal Medical Models
AI总结：
针对医学多模态大模型缺乏开源评估器的问题，提出ACE-$M^3$，采用分支-合并架构和RTDPO策略，实现高效准确的自动能力评估。
摘要：
随着多模态大语言模型（MLLMs）在医学领域日益受到重视，开发精确的评估方法来衡量其有效性的需求变得至关重要。虽然基准测试为评估MLLMs的能力提供了可靠手段，但用于开放领域评估的传统指标（如ROUGE和BLEU）仅关注词元重叠，可能无法与人类判断保持一致。尽管人工评估更为可靠，但它劳动密集、成本高昂且不可扩展。基于LLM的评估方法已被证明有前景，但迄今为止，医学领域仍迫切需要开源的基于多模态LLM的评估器。为解决这一问题，我们推出了ACE-$M^3$，一个开源的自动能力评估器，专门用于评估医学MLLMs的问答能力。它首先采用分支-合并架构，基于标准医学评估标准提供详细分析和简洁的最终评分。随后，引入基于奖励令牌的直接偏好优化（RTDPO）策略，以节省训练时间且不损害模型性能。大量实验证明了我们的ACE-$M^3$模型在评估医学MLLMs能力方面的有效性。
英文摘要：
As multimodal large language models (MLLMs) gain prominence in the medical field, the need for precise evaluation methods to assess their effectiveness has become critical. While benchmarks provide a reliable means to evaluate the capabilities of MLLMs, traditional metrics like ROUGE and BLEU employed for open domain evaluation only focus on token overlap and may not align with human judgment. Although human evaluation is more reliable, it is labor-intensive, costly, and not scalable. LLM-based evaluation methods have proven promising, but to date, there is still an urgent need for open-source multimodal LLM-based evaluators in the medical field. To address this issue, we introduce ACE-$M^3$, an open-sourced \textbf{A}utomatic \textbf{C}apability \textbf{E}valuator for \textbf{M}ultimodal \textbf{M}edical \textbf{M}odels specifically designed to assess the question answering abilities of medical MLLMs. It first utilizes a branch-merge architecture to provide both detailed analysis and a concise final score based on standard medical evaluation criteria. Subsequently, a reward token-based direct preference optimization (RTDPO) strategy is incorporated to save training time without compromising performance of our model. Extensive experiments have demonstrated the effectiveness of our ACE-$M^3$ model\footnote{\url{https://huggingface.co/collections/AIUSRTMP/ace-m3-67593297ff391b93e3e5d068}} in evaluating the capabilities of medical MLLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
