arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.08985cs.CV

LLaVA-UHD v4: 什么使多模态大语言模型中的高效视觉编码成为可能？

LLaVA-UHD v4: What Makes Efficient Visual Encoding in MLLMs?

Tsinghua University（清华大学）；ModelBest

Kechen Fang, Yihua Qin, Chongyi Wang, Wenshuo Ma, Tianyu Yu, Yuan Yao

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文通过 slice-based 编码和 intra-ViT 早期压缩，提升了多模态大语言模型在高分辨率图像输入中的视觉编码效率，减少 55.8% 的 FLOPs 而不牺牲下游性能。

摘要：

视觉编码是多模态大语言模型（MLLMs）中的主要计算瓶颈，尤其在高分辨率图像输入中更为明显。现有方法通常采用全局编码后接 ViT 压缩。全局编码会产生大量 token 序列，而 ViT 压缩则在任何 token 减少之前都需要完整的二次注意力成本。在本工作中，我们从编码策略和视觉 token 压缩两个维度重新审视这一传统。首先，受控实验表明，基于切片的编码在基准测试中优于全局编码，表明通过切片视图保留局部细节比应用全局注意力进行细粒度感知更有效。其次，我们引入了 intra-ViT 早期压缩，通过在浅层 ViT 层中减少 token，显著降低了视觉编码的 FLOPs 而又保持了下游性能。通过将 intra-ViT 压缩整合到基于切片的编码框架中，我们提出了 LLaVA-UHD v4，一种针对高分辨率输入的高效且计算可控的视觉编码方案。在涵盖文档理解、OCR 和通用 VQA 的多样化基准测试中，LLaVA-UHD v4 将视觉编码的 FLOPs 减少了 55.8%，同时匹配或超越了基线性能。这些结果表明，视觉编码效率可以显著提高，而不牺牲下游性能，为高效高分辨率 MLLMs 提供了实用的设计方向。所有模型权重和代码将公开发布以支持进一步研究。

英文摘要：

Visual encoding constitutes a major computational bottleneck in Multimodal Large Language Models (MLLMs), especially for high-resolution image inputs. The prevailing practice typically adopts global encoding followed by post-ViT compression. Global encoding produces massive token sequences, while post-ViT compression incurs the full quadratic attention cost of the ViT before any token reduction takes place. In this work, we revisit this convention along two dimensions: the encoding strategy and visual token compression. First, controlled experiments show that slice-based encoding outperforms global encoding across benchmarks, suggesting that preserving local details through sliced views can be more beneficial than applying global attention for fine-grained perception. Second, we introduce intra-ViT early compression, which reduces tokens in shallow ViT layers and substantially lowers visual-encoding FLOPs while preserving downstream performance. By integrating intra-ViT compression into the slice-based encoding framework, we present LLaVA-UHD v4, an efficient and compute-controllable visual encoding scheme tailored for high-resolution inputs. Across a diverse set of benchmarks covering document understanding, OCR, and general VQA, LLaVA-UHD v4 reduces visual-encoding FLOPs by 55.8% while matching or even surpassing baseline performance. These results suggest that visual-encoding efficiency can be substantially improved without sacrificing downstream performance, providing a practical design direction for efficient high-resolution MLLMs. All model weights and code will be publicly released to support further research.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑