arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.17606cs.CV
收藏

HyperSeg:利用大型语言模型迈向通用视觉分割

HyperSeg: Towards Universal Visual Segmentation with Large Language Model

Tsinghua University(清华大学);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);Meituan Inc.(美团公司)

Cong Wei, Yujie Zhong, Haoxian Tan, Yong Liu, Zheng Zhao, Jie Hu, Yujiu Yang

AI总结:

本文提出首个基于 VLLM 的像素级图像与视频通用分割模型 HyperSeg,通过混合实体识别、细粒度视觉感知器和时序适配器增强复杂推理分割能力。

摘要:

本文旨在借助视觉大型语言模型(VLLM)所赋予的强大推理能力,解决面向图像与视频感知的通用分割问题。尽管当前统一分割方法已取得显著进展,但它们在同时适配图像和视频场景方面存在局限,且难以处理复杂的推理分割,因此难以应对各类具有挑战性的指令,也无法准确理解细粒度的视觉-语言关联。我们提出 HyperSeg,这是首个基于 VLLM 的通用分割模型,用于像素级图像和视频感知,涵盖通用分割任务以及需要强大推理能力和世界知识的更复杂推理感知任务。此外,为充分利用 VLLM 的识别能力和细粒度视觉信息,HyperSeg 融合了混合实体识别模块与细粒度视觉感知器模块,以处理各类分割任务。结合时序适配器,HyperSeg 实现了对时序信息的全面理解。实验结果验证了我们的思路在解决通用图像和视频分割任务(包括更复杂的推理感知任务)方面的有效性。我们的代码已公开。

英文摘要:

This paper aims to address universal segmentation for image and video perception with the strong reasoning ability empowered by Visual Large Language Models (VLLMs). Despite significant progress in current unified segmentation methods, limitations in adaptation to both image and video scenarios, as well as the complex reasoning segmentation, make it difficult for them to handle various challenging instructions and achieve an accurate understanding of fine-grained vision-language correlations. We propose HyperSeg, the first VLLM-based universal segmentation model for pixel-level image and video perception, encompassing generic segmentation tasks and more complex reasoning perception tasks requiring powerful reasoning abilities and world knowledge. Besides, to fully leverage the recognition capabilities of VLLMs and the fine-grained visual information, HyperSeg incorporates hybrid entity recognition and fine-grained visual perceiver modules for various segmentation tasks. Combined with the temporal adapter, HyperSeg achieves a comprehensive understanding of temporal information. Experimental results validate the effectiveness of our insights in resolving universal image and video segmentation tasks, including the more complex reasoning perception tasks. Our code is available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑