arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.07783cs.CVcs.CL
收藏

用于视觉感知与多模态理解的参数倒置图像金字塔网络

Parameter-Inverted Image Pyramid Networks for Visual Perception and Multimodal Understanding

Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Sensetime

Zhaokai Wang, Xizhou Zhu, Xue Yang, Gen Luo, Hao Li, Changyao Tian, Wenhan Dou, Junqi Ge, Lewei Lu, Yu Qiao, Jifeng Dai

AI总结:

提出 PIIP 网络,以小模型处理高分辨率图像并进行跨分支特征交互,在降低计算成本的同时提升视觉感知与多模态理解性能。

摘要:

图像金字塔被性能最优的方法广泛采用,以获取多尺度特征,从而实现精确的视觉感知与理解。然而,当前图像金字塔使用同一个大规模模型处理多种分辨率的图像,导致显著的计算成本。为应对这一挑战,我们提出一种名为 Parameter-Inverted Image Pyramid Networks(PIIP,参数倒置图像金字塔网络)的新型网络架构。具体而言,PIIP 使用预训练模型(ViT 或 CNN)作为分支来处理多尺度图像,其中更高分辨率的图像由更小的网络分支处理,以平衡计算成本与性能。为整合来自不同空间尺度的信息,我们进一步提出一种新颖的跨分支特征交互机制。为验证 PIIP,我们将其应用于多种感知模型以及一个具有代表性的多模态大语言模型 LLaVA,并在目标检测、分割、图像分类和多模态理解等多种任务上开展大量实验。与单分支方法和现有多分辨率方法相比,PIIP 以更低的计算成本取得了更优性能。当应用于大规模视觉基础模型 InternViT-6B 时,PIIP 仅用原有计算量的 40%-60%,即可将其在检测和分割上的性能提升 1%-2%,最终在 MS COCO 上达到 60.0 box AP,在 ADE20K 上达到 59.7 mIoU。在多模态理解方面,我们的 PIIP-LLaVA 仅使用 280 万条训练数据,就在 TextVQA 上取得 73.0% 的准确率,并在 MMBench 上取得 74.5% 的准确率。我们的代码已发布于 https://github.com/OpenGVLab/PIIP。

英文摘要:

Image pyramids are widely adopted in top-performing methods to obtain multi-scale features for precise visual perception and understanding. However, current image pyramids use the same large-scale model to process multiple resolutions of images, leading to significant computational cost. To address this challenge, we propose a novel network architecture, called Parameter-Inverted Image Pyramid Networks (PIIP). Specifically, PIIP uses pretrained models (ViTs or CNNs) as branches to process multi-scale images, where images of higher resolutions are processed by smaller network branches to balance computational cost and performance. To integrate information from different spatial scales, we further propose a novel cross-branch feature interaction mechanism. To validate PIIP, we apply it to various perception models and a representative multimodal large language model called LLaVA, and conduct extensive experiments on various tasks such as object detection, segmentation, image classification and multimodal understanding. PIIP achieves superior performance compared to single-branch and existing multi-resolution approaches with lower computational cost. When applied to InternViT-6B, a large-scale vision foundation model, PIIP can improve its performance by 1%-2% on detection and segmentation with only 40%-60% of the original computation, finally achieving 60.0 box AP on MS COCO and 59.7 mIoU on ADE20K. For multimodal understanding, our PIIP-LLaVA achieves 73.0% accuracy on TextVQA and 74.5% on MMBench with only 2.8M training data. Our code is released at https://github.com/OpenGVLab/PIIP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑