arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.02034cs.CV
收藏

Mini-Monkey:通过互补图像金字塔缓解轻量级 MLLM 的语义锯齿效应

Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramid

Huazhong University of Science and Technology(华中科技大学);South China University of Technology(华南理工大学)

Mingxin Huang, Yuliang Liu, Dingkang Liang, Lianwen Jin, Xiang Bai

AI总结:

本文提出即插即用的互补图像金字塔 CIP 与尺度压缩机制 SCM,缓解高分辨率裁剪造成的语义锯齿效应,并据此构建轻量级 MLLM Mini-Monkey,在多模态与文档理解上表现优异。

摘要:

近年来,将图像缩放到高分辨率在多模态大语言模型(MLLM)中受到广泛关注。现有大多数方法采用滑动窗口式裁剪策略来适应分辨率提升。然而,这种裁剪策略很容易切断物体和连通区域,从而引入语义不连续,妨碍 MLLM 识别小尺寸或形状不规则的物体或文本,导致一种我们称为语义锯齿效应的现象。该效应在轻量级 MLLM 中尤为明显。为解决这一问题,我们提出互补图像金字塔(CIP),这是一种简单、有效且即插即用的方案,旨在缓解高分辨率图像处理过程中的语义不连续。具体而言,CIP 动态构建图像金字塔,为基于裁剪的 MLLM 提供互补语义信息,使其能够充分获取各个层级的语义。此外,我们提出尺度压缩机制(SCM),通过压缩冗余视觉 token 来减少额外计算开销。实验表明,CIP 能够在多种架构(如 MiniCPM-V-2、InternVL2 和 LLaVA-OneVision)、不同模型容量(1B→8B)以及不同使用配置(免训练和微调)下持续提升性能。利用所提出的 CIP 和 SCM,我们推出轻量级 MLLM Mini-Monkey,其在通用多模态理解和文档理解方面均取得出色性能。在 OCRBench 上,2B 版本的 Mini-Monkey 甚至比 8B 模型 InternVL2-8B 高出 12 分。此外,训练 Mini-Monkey 成本很低,仅需八张 RTX 3090 GPU。代码可在 https://github.com/Yuliang-Liu/Monkey 获取。

英文摘要:

Recently, scaling images to high resolution has received much attention in multimodal large language models (MLLMs). Most existing practices adopt a sliding-window-style cropping strategy to adapt to resolution increase. Such a cropping strategy, however, can easily cut off objects and connected regions, which introduces semantic discontinuity and therefore impedes MLLMs from recognizing small or irregularly shaped objects or text, leading to a phenomenon we call the semantic sawtooth effect. This effect is particularly evident in lightweight MLLMs. To address this issue, we introduce a Complementary Image Pyramid (CIP), a simple, effective, and plug-and-play solution designed to mitigate semantic discontinuity during high-resolution image processing. In particular, CIP dynamically constructs an image pyramid to provide complementary semantic information for the cropping-based MLLMs, enabling them to richly acquire semantics at all levels. Furthermore, we introduce a Scale Compression Mechanism (SCM) to reduce the additional computational overhead by compressing the redundant visual tokens. Our experiments demonstrate that CIP can consistently enhance the performance across diverse architectures (e.g., MiniCPM-V-2, InternVL2, and LLaVA-OneVision), various model capacity (1B$\rightarrow$8B), and different usage configurations (training-free and fine-tuning). Leveraging the proposed CIP and SCM, we introduce a lightweight MLLM, Mini-Monkey, which achieves remarkable performance in both general multimodal understanding and document understanding. On the OCRBench, the 2B-version Mini-Monkey even surpasses the 8B model InternVL2-8B by 12 score. Additionally, training Mini-Monkey is cheap, requiring only eight RTX 3090 GPUs. The code is available at https://github.com/Yuliang-Liu/Monkey.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑