arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.12191cs.CVcs.AIcs.CL
收藏

Qwen2-VL:增强视觉语言模型在任意分辨率下对世界的感知能力

Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

Alibaba Group(阿里巴巴集团)
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng … 展开作者

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, Junyang Lin

AI总结:

Qwen2-VL系列通过Naive Dynamic Resolution机制和M-RoPE,统一处理图像与视频,并借助缩放定律在2B至72B参数规模上取得与GPT-4o等领先模型相当的多模态性能。

摘要:

我们提出了Qwen2-VL系列,这是对先前Qwen-VL模型的高级升级,重新定义了视觉处理中传统的预定分辨率方法。Qwen2-VL引入了Naive Dynamic Resolution机制,使模型能够动态地将不同分辨率的图像处理为不同数量的视觉token。这种方法使模型能够生成更高效、更准确的视觉表示,与人类的感知过程紧密对齐。该模型还集成了多模态旋转位置嵌入(M-RoPE),促进了文本、图像和视频之间位置信息的有效融合。我们采用统一的范式来处理图像和视频,增强了模型的视觉感知能力。为了探索大型多模态模型的潜力,Qwen2-VL研究了大型视觉语言模型(LVLMs)的缩放定律。通过同时扩展模型规模——提供2B、8B和72B参数的版本——以及训练数据量,Qwen2-VL系列取得了极具竞争力的性能。值得注意的是,Qwen2-VL-72B模型在多个多模态基准测试上取得了与GPT-4o和Claude3.5-Sonnet等领先模型相当的结果,超越了其他通用模型。代码可在https://github.com/QwenLM/Qwen2-VL获取。

英文摘要:

We present the Qwen2-VL Series, an advanced upgrade of the previous Qwen-VL models that redefines the conventional predetermined-resolution approach in visual processing. Qwen2-VL introduces the Naive Dynamic Resolution mechanism, which enables the model to dynamically process images of varying resolutions into different numbers of visual tokens. This approach allows the model to generate more efficient and accurate visual representations, closely aligning with human perceptual processes. The model also integrates Multimodal Rotary Position Embedding (M-RoPE), facilitating the effective fusion of positional information across text, images, and videos. We employ a unified paradigm for processing both images and videos, enhancing the model's visual perception capabilities. To explore the potential of large multimodal models, Qwen2-VL investigates the scaling laws for large vision-language models (LVLMs). By scaling both the model size-with versions at 2B, 8B, and 72B parameters-and the amount of training data, the Qwen2-VL Series achieves highly competitive performance. Notably, the Qwen2-VL-72B model achieves results comparable to leading models such as GPT-4o and Claude3.5-Sonnet across various multimodal benchmarks, outperforming other generalist models. Code is available at https://github.com/QwenLM/Qwen2-VL .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑