arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.14035cs.CLcs.AI
收藏

利用游戏玩法研究大型多模态模型中的多模态与对话式基础

Using Game Play to Investigate Multimodal and Conversational Grounding in Large Multimodal Models

University of Potsdam(波茨坦大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心)

Sherzod Hakimov, Yerkezhan Abdullayeva, Kushal Koshti, Antonia Schmidt, Yan Weiser, Anne Beyer, David Schlangen

AI总结:

针对多模态模型评估方法滞后于模型发展的问题,本文将目标导向型游戏自我博弈的评估范式引入多模态领域,设计游戏测试视觉表征与对话对齐能力,发现闭源大模型表现优于开源模型,且深度字幕能力是关键影响因素,该基准仍具研究价值。

摘要:

尽管纯文本模型的评估状况已有所改善,但目前多模态(文本与图像)模型的发展速度似乎又快于其评估方法的迭代速度。本文将文本模型领域近期提出的一种评估范式——即通过目标导向型游戏(自我)博弈进行评估——引入多模态模型领域,以此补充基于参考的评估与基于偏好的评估方法。具体而言,我们定义了若干游戏,用于测试模型从视觉信息中表征场景、并通过对话对齐此类表征的能力。研究发现,规模最大的闭源模型在我们定义的游戏中表现相当出色,而即便是性能最优的开源权重模型也难以应对这些游戏。进一步分析显示,最大规模模型出众的深度字幕能力是其部分性能的驱动因素。两类模型仍有提升空间,这也确保了该基准的持续研究价值。

英文摘要:

While the situation has improved for text-only models, it again seems to be the case currently that multimodal (text and image) models develop faster than ways to evaluate them. In this paper, we bring a recently developed evaluation paradigm from text models to multimodal models, namely evaluation through the goal-oriented game (self) play, complementing reference-based and preference-based evaluation. Specifically, we define games that challenge a model's capability to represent a situation from visual information and align such representations through dialogue. We find that the largest closed models perform rather well on the games that we define, while even the best open-weight models struggle with them. On further analysis, we find that the exceptional deep captioning capabilities of the largest models drive some of the performance. There is still room to grow for both kinds of models, ensuring the continued relevance of the benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑