arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.07022cs.CLcs.AIcs.CV
收藏

ViLMA:视频-语言模型中语言与时间定位的零样本基准

ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models

Koç University(科奇大学);KUIS AI Center(KUIS人工智能中心);University of Pisa(比萨大学);Hacettepe University(哈塞特佩大学);Aselsan Research(阿塞尔桑研究);University of Malta(马耳他大学);Heidelberg University(海德堡大学);Amsterdam UMC, University of Amsterdam(阿姆斯特丹大学医学中心);Utrecht University(乌得勒支大学)

Ilker Kesen, Andrea Pedrotti, Mustafa Dogan, Michele Cafagna, Emre Can Acikgoz, Letitia Parcalabescu, Iacer Calixto, Anette Frank, Albert Gatt, Aykut Erdem, Erkut Erdem

AI总结:

ViLMA是一个任务无关的零样本基准,通过反事实和熟练度测试评估视频-语言模型的细粒度能力,揭示其定位能力不优于静态图像模型,为未来研究指明方向。

摘要:

随着预训练视频-语言模型(VidLMs)日益流行,迫切需要开发稳健的评估方法,以更深入地探究其视觉-语言能力。为应对这一挑战,我们提出了ViLMA(视频语言模型评估),一个任务无关的基准,为评估这些模型的细粒度能力奠定了坚实基础。基于任务的评估虽然有价值,但无法捕捉VidLMs需要处理的动态图像中的复杂性和特定时间方面。通过精心设计的反事实,ViLMA提供了一个受控评估套件,揭示了这些模型的真实潜力及其与人类水平理解相比的性能差距。ViLMA还包括熟练度测试,评估被认为对解决主要反事实测试至关重要的基本能力。我们表明,当前VidLMs的定位能力并不优于使用静态图像的视觉-语言模型。一旦将熟练度测试的性能考虑在内,这一点尤其引人注目。我们的基准作为未来VidLMs研究的催化剂,有助于突出仍需探索的领域。

英文摘要:

With the ever-increasing popularity of pretrained Video-Language Models (VidLMs), there is a pressing need to develop robust evaluation methodologies that delve deeper into their visio-linguistic capabilities. To address this challenge, we present ViLMA (Video Language Model Assessment), a task-agnostic benchmark that places the assessment of fine-grained capabilities of these models on a firm footing. Task-based evaluations, while valuable, fail to capture the complexities and specific temporal aspects of moving images that VidLMs need to process. Through carefully curated counterfactuals, ViLMA offers a controlled evaluation suite that sheds light on the true potential of these models, as well as their performance gaps compared to human-level understanding. ViLMA also includes proficiency tests, which assess basic capabilities deemed essential to solving the main counterfactual tests. We show that current VidLMs' grounding abilities are no better than those of vision-language models which use static images. This is especially striking once the performance on proficiency tests is factored in. Our benchmark serves as a catalyst for future research on VidLMs, helping to highlight areas that still need to be explored.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑