RSUniVLM：通过面向粒度的混合专家构建的遥感统一视觉语言模型
RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Experts
AI总结：
本文提出约10亿参数的统一端到端遥感视觉语言模型 RSUniVLM，通过面向粒度的混合专家架构支持图像、区域、像素级及多图像任务，并构建大规模指令数据集验证其先进性能。
摘要：
遥感视觉语言模型（RS VLM）在遥感（RS）图像理解任务上已取得显著进展。尽管现有模型在多模态推理和多轮对话中表现良好，但缺乏像素级理解能力，并且难以处理多图像输入。在本研究中，我们提出 RSUniVLM，这是一个统一的端到端 RS VLM，旨在实现跨多种粒度的全面视觉理解，包括图像级、区域级和像素级任务。RSUniVLM 还能有效执行多图像分析，例如变化检测和变化描述等实例。为了在不增加模型规模的情况下增强模型捕获不同层级视觉信息的能力，我们设计了一种名为 Granularity-oriented Mixture of Experts（面向粒度的混合专家）的新型架构，将模型参数量限制在约 10 亿。我们还基于遥感和通用领域的多种现有数据集，构建了一个大规模遥感指令遵循数据集，涵盖目标定位、视觉问答和语义分割等各类任务。我们开展了大量实验，验证所提出的 RSUniVLM 在各类遥感任务上达到最先进水平的优越性。代码和模型将在 https://github.com/xuliu-cyber/RSUniVLM 提供。
英文摘要：
Remote Sensing Vision-Language Models (RS VLMs) have made much progress in the tasks of remote sensing (RS) image comprehension. While performing well in multi-modal reasoning and multi-turn conversations, the existing models lack pixel-level understanding and struggle with multi-image inputs. In this work, we propose RSUniVLM, a unified, end-to-end RS VLM designed for comprehensive vision understanding across multiple granularity, including image-level, region-level, and pixel-level tasks. RSUniVLM also performs effectively in multi-image analysis, with instances of change detection and change captioning. To enhance the model's ability to capture visual information at different levels without increasing model size, we design a novel architecture called Granularity-oriented Mixture of Experts to constraint the model to about 1 billion parameters. We also construct a large-scale RS instruction-following dataset based on a variety of existing datasets in both RS and general domain, encompassing various tasks such as object localization, visual question answering, and semantic segmentation. Substantial experiments have been conducted to validate the superiority of the proposed RSUniVLM up to state-of-the-art across various RS tasks. Code and model will be available at \href{https://github.com/xuliu-cyber/RSUniVLM}{here}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
