GeoPix：用于遥感中像素级图像理解的多模态大语言模型
GeoPix: Multi-Modal Large Language Model for Pixel-level Image Understanding in Remote Sensing
AI总结：
针对遥感MLLMs缺乏像素级对话能力的问题，本文提出GeoPix，通过掩码预测器、类别级可学习记忆、GeoPixInstruct数据集和两阶段训练实现像素级分割，并保持图像及区域级任务竞争力。
摘要：
多模态大语言模型（MLLMs）已在图像级和区域级遥感（RS）图像理解任务中取得显著成功，例如图像描述、视觉问答和视觉定位。然而，现有遥感MLLMs缺乏像素级对话能力，该能力涉及利用针对特定实例的分割掩码来响应用户指令。本文提出GeoPix，这是一种将图像理解能力扩展至像素级的遥感MLLM。实现方式是为MLLM配备一个掩码预测器，该预测器以LLM的分割token嵌入为条件，将来自视觉编码器的视觉特征转换为掩码。为便于分割遥感图像中的多尺度目标，掩码预测器中集成了一个类别级可学习记忆模块，用于在整个数据集范围内捕获并存储实例级的类别级地理上下文。此外，为解决训练像素级遥感MLLMs所需大规模数据集缺失的问题，我们构建了GeoPixInstruct数据集，包含65,463张图像和140,412个实例，每个实例均标注有文本描述、边界框和掩码。而且，我们制定了两阶段训练策略，以在多模态多任务优化中平衡文本生成与掩码预测的不同需求。大量实验验证了GeoPix在像素级分割任务中的有效性和优越性，同时其在图像级和区域级基准测试中也保持了具有竞争力的性能。
英文摘要：
Multi-modal large language models (MLLMs) have achieved remarkable success in image- and region-level remote sensing (RS) image understanding tasks, such as image captioning, visual question answering, and visual grounding. However, existing RS MLLMs lack the pixel-level dialogue capability, which involves responding to user instructions with segmentation masks for specific instances. In this paper, we propose GeoPix, a RS MLLM that extends image understanding capabilities to the pixel level. This is achieved by equipping the MLLM with a mask predictor, which transforms visual features from the vision encoder into masks conditioned on the LLM's segmentation token embeddings. To facilitate the segmentation of multi-scale objects in RS imagery, a class-wise learnable memory module is integrated into the mask predictor to capture and store class-wise geo-context at the instance level across the entire dataset. In addition, to address the absence of large-scale datasets for training pixel-level RS MLLMs, we construct the GeoPixInstruct dataset, comprising 65,463 images and 140,412 instances, with each instance annotated with text descriptions, bounding boxes, and masks. Furthermore, we develop a two-stage training strategy to balance the distinct requirements of text generation and masks prediction in multi-modal multi-task optimization. Extensive experiments verify the effectiveness and superiority of GeoPix in pixel-level segmentation tasks, while also maintaining competitive performance in image- and region-level benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
