arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.16500cs.CV
收藏

CogVLM2:用于图像与视频理解的视觉语言模型

CogVLM2: Visual Language Models for Image and Video Understanding

Zhipu AI(智谱AI);Tsinghua University(清华大学)
Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, Lei Zhao, Zhuoyi Yang, Xiaotao Gu, Xiaohan … 展开作者

Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, Lei Zhao, Zhuoyi Yang, Xiaotao Gu, Xiaohan Zhang, Guanyu Feng, Da Yin, Zihan Wang, Ji Qi, Xixuan Song, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Yuxiao Dong, Jie Tang

AI总结:

本文提出 CogVLM2 系列视觉语言模型,涵盖图像与视频理解,通过改进训练方法、支持高分辨率输入和自动化时序定位数据构建,在多个基准上取得领先结果并全面开源。

摘要:

从 VisualGLM 和 CogVLM 开始,我们持续探索视觉语言模型(VLM),以追求更强的视觉-语言融合、更高效的高分辨率架构以及更广泛的模态和应用。在此,我们提出 CogVLM2 系列,这是用于图像与视频理解的新一代视觉语言模型,包括 CogVLM2、CogVLM2-Video 和 GLM-4V。作为图像理解模型,CogVLM2 继承了视觉专家架构,并在预训练和后训练阶段改进了训练方法,支持高达 $1344 \ imes 1344$ 像素的输入分辨率。作为视频理解模型,CogVLM2-Video 将多帧输入与时间戳相结合,并提出了自动化的时序定位数据构建方法。值得注意的是,CogVLM2 系列在 MMBench、MM-Vet、TextVQA、MVBench 和 VCGBench 等基准上取得了最先进的结果。所有模型均在 https://github.com/THUDM/CogVLM2 和 https://github.com/THUDM/GLM-4 开源,为该领域的发展做出了贡献。

英文摘要:

Beginning with VisualGLM and CogVLM, we are continuously exploring VLMs in pursuit of enhanced vision-language fusion, efficient higher-resolution architecture, and broader modalities and applications. Here we propose the CogVLM2 family, a new generation of visual language models for image and video understanding including CogVLM2, CogVLM2-Video and GLM-4V. As an image understanding model, CogVLM2 inherits the visual expert architecture with improved training recipes in both pre-training and post-training stages, supporting input resolution up to $1344 \times 1344$ pixels. As a video understanding model, CogVLM2-Video integrates multi-frame input with timestamps and proposes automated temporal grounding data construction. Notably, CogVLM2 family has achieved state-of-the-art results on benchmarks like MMBench, MM-Vet, TextVQA, MVBench and VCGBench. All models are open-sourced in https://github.com/THUDM/CogVLM2 and https://github.com/THUDM/GLM-4, contributing to the advancement of the field.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑