arXiv 2506.168062025-09-21cs.CV

FOCUS：面向由指代分割驱动的交互式编辑的统一视觉-语言建模

FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation

Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Science（中国科学院大学人工智能学院）；Peng Cheng Laboratory, Shenzhen, China（鹏城实验室）；Wuhan AI Research, Wuhan, China（武汉人工智能研究所）

Fan Yang, Yousong Zhu, Xin Li, Yufei Zhan, Hongyin Zhao, Shurong Zheng, Yaowei Wang, Ming Tang, Jinqiao Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

FOCUS提出统一视觉-语言模型，通过双分支编码器、MoVQGAN视觉分词与渐进式多阶段训练，将分割感知和可控目标生成整合到端到端框架中，提升多模态理解、指代分割与图像编辑性能。

摘要：

近期的大型视觉语言模型（LVLMs）在统一视觉理解与生成建模方面展现出令人期待的能力，既能实现准确的内容理解，也能进行灵活的编辑。然而，当前方法将“看什么”与“如何编辑”分开处理：它们要么执行孤立的目标分割，要么仅将分割掩码用作局部编辑生成任务的条件提示，通常依赖多个彼此割裂的模型。为弥合这些差距，我们提出了FOCUS，一个统一的LVLM，在端到端框架内整合了分割感知的感知能力与可控的以目标为中心的生成能力。FOCUS采用双分支视觉编码器，同时捕获全局语义上下文和细粒度空间细节。此外，我们利用基于MoVQGAN的视觉分词器生成离散视觉标记，以提升生成质量。为了实现准确且可控的图像编辑，我们提出了一种渐进式多阶段训练流程，其中分割掩码被联合优化，并作为空间条件提示来引导扩散解码器。该策略对齐了视觉编码、分割和生成模块，有效地将分割感知的感知能力与细粒度视觉合成连接起来。在包括多模态理解、指代分割精度和可控图像生成在内的三项核心任务上的大量实验表明，FOCUS通过联合优化视觉感知与生成能力，取得了强劲的性能。

英文摘要：

Recent Large Vision Language Models (LVLMs) demonstrate promising capabilities in unifying visual understanding and generative modeling, enabling both accurate content understanding and flexible editing. However, current approaches treat "what to see" and "how to edit" separately: they either perform isolated object segmentation or utilize segmentation masks merely as conditional prompts for local edit generation tasks, often relying on multiple disjointed models. To bridge these gaps, we introduce FOCUS, a unified LVLM that integrates segmentation-aware perception and controllable object-centric generation within an end-to-end framework. FOCUS employs a dual-branch visual encoder to simultaneously capture global semantic context and fine-grained spatial details. In addition, we leverage a MoVQGAN-based visual tokenizer to produce discrete visual tokens that enhance generation quality. To enable accurate and controllable image editing, we propose a progressive multi-stage training pipeline, where segmentation masks are jointly optimized and used as spatial condition prompts to guide the diffusion decoder. This strategy aligns visual encoding, segmentation, and generation modules, effectively bridging segmentation-aware perception with fine-grained visual synthesis. Extensive experiments across three core tasks, including multimodal understanding, referring segmentation accuracy, and controllable image generation, demonstrate that FOCUS achieves strong performance by jointly optimizing visual perception and generative capabilities.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。