arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.13951cs.CL
收藏

MLLM-Bench:基于逐样本准则评估多模态大语言模型

MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria

Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
Wentao Ge, Shunian Chen, Guiming Hardy Chen, Junying Chen, Zhihong Chen, Nuo Chen, Wenya Xie, Shuo Yan, Chenghao Zhu, Ziyue Lin, Song Dingjie, Xidong Wang, Anni… 展开作者

Wentao Ge, Shunian Chen, Guiming Hardy Chen, Junying Chen, Zhihong Chen, Nuo Chen, Wenya Xie, Shuo Yan, Chenghao Zhu, Ziyue Lin, Song Dingjie, Xidong Wang, Anningzhe Gao, Zhang Zhiyi, Jianquan Li, Xiang Wan, Benyou Wang

AI总结:

针对现有MLLM评估方法难以适配开放主观类多模态任务的问题,提出基于逐样本准则、用强MLLM做评判的评估范式,构建了覆盖六个认知层级的MLLM-Bench基准,测试21个模型且与人类评估一致性达88.02%。

摘要:

多模态大语言模型(MLLMs)拓展了AI应用的范围。现有的MLLM自动评估方法主要局限于评估查询时未考虑用户体验,无法充分应对创意性和联想性多模态任务的细微差异。然而,这类任务的开放性和主观性给评估方法带来了重大挑战,即很难为其定义标准答案。为此,本文提出了一种新的MLLM评估范式:使用强大的MLLM作为评判者,基于逐样本准则对MLLMs进行评估。为验证该范式的可行性与有效性,我们设计了名为MLLM-Bench的基准,通过整理覆盖六个综合认知层级的评估样本构建而成。我们以成对比较的方式对21个主流MLLM进行了基准测试,结果显示不同模型的表现各异。此外,该基准的有效性体现在与人类评估的一致性达到88.02%。我们认为,所提出的范式借助逐样本准则,挖掘了MLLMs作为有效评估工具的潜力。在线排行榜见https://mllm-bench.llmzoo.com。

英文摘要:

Multimodal large language models (MLLMs) have broadened the scope of AI applications. Existing automatic evaluation methodologies for MLLMs are mainly limited in evaluating queries without considering user experiences, inadequately addressing the nuances of creative and associative multimodal tasks. However, the open-ended and subjective nature of such tasks poses a significant challenge to the evaluation methodology, where it is difficult to define the ground-truth answers for them. To this end, in our paper, we propose a new evaluation paradigm for MLLMs, which is evaluating MLLMs with per-sample criteria using potent MLLM as the judge. To validate the feasibility and effectiveness of this paradigm, we design a benchmark, dubbed MLLM-Bench, by curating the evaluation samples across six comprehensive cognitive levels. We benchmark 21 popular MLLMs in a pairwise-comparison fashion, showing diverse performance across models. Moreover, the validity of our benchmark manifests itself in reaching 88.02% agreement with human evaluation. We contend that the proposed paradigm explores the potential of MLLMs as effective evaluation tools with the help of per-sample criteria. See online leaderboard at \url{https://mllm-bench.llmzoo.com}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑