arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.00522cs.CV
收藏

VisionLLaMA:面向视觉任务的统一 LLaMA 骨干网络

VisionLLaMA: A Unified LLaMA Backbone for Vision Tasks

Meituan Inc.(美团公司);Moonshot AI(月之暗面人工智能公司);Zhejiang University(浙江大学)

Xiangxiang Chu, Jianlin Su, Bo Zhang, Chunhua Shen

AI总结:

本文提出统一的类 LLaMA 视觉骨干网络 VisionLLaMA,以普通和金字塔形式适配二维图像,并在感知与生成任务上验证其优于现有 ViT 的性能。

摘要:

大型语言模型建立在基于 transformer 的架构之上,以处理文本输入。例如,LLaMA 在众多开源实现中表现突出。同一个 transformer 能否用于处理二维图像?在本文中,我们通过揭示一种为此定制的、具有普通形式和金字塔形式的类 LLaMA 视觉 transformer(称为 VisionLLaMA)来回答这一问题。VisionLLaMA 是一个统一且通用的建模框架,用于解决大多数视觉任务。我们在大量下游图像感知任务,尤其是图像生成任务中,使用典型的预训练范式广泛评估了其有效性。在许多情况下,VisionLLaMA 相比此前最先进的视觉 transformer 展现出显著提升。我们相信 VisionLLaMA 可以作为视觉生成与理解的一个强大的新型基线模型。我们的代码发布于 https://github.com/Meituan-AutoML/VisionLLaMA。

英文摘要:

Large language models are built on top of a transformer-based architecture to process textual inputs. For example, the LLaMA stands out among many open-source implementations. Can the same transformer be used to process 2D images? In this paper, we answer this question by unveiling a LLaMA-like vision transformer in plain and pyramid forms, termed VisionLLaMA, which is tailored for this purpose. VisionLLaMA is a unified and generic modelling framework for solving most vision tasks. We extensively evaluate its effectiveness using typical pre-training paradigms in a good portion of downstream tasks of image perception and especially image generation. In many cases, VisionLLaMA have exhibited substantial gains over the previous state-of-the-art vision transformers. We believe that VisionLLaMA can serve as a strong new baseline model for vision generation and understanding. Our code is released at https://github.com/Meituan-AutoML/VisionLLaMA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑