基于指令增强大语言模型的光谱蒸馏表示用于卫星图像
Spectrally Distilled Representations Aligned with Instruction-Augmented LLMs for Satellite Imagery
AI总结:
SATtxt通过光谱蒸馏和指令增强大语言模型实现卫星图像的光谱感知视觉-语言学习,提升了零样本分类、检索和线性探测性能。
摘要:
视觉-语言基础模型(VLFMs)承诺为地球观测提供零样本和检索理解。尽管运营卫星系统往往缺乏完整的多光谱覆盖,使仅使用RGB进行推理成为可扩展部署的首选,但将VLFMs应用于卫星图像仍受到两个因素的阻碍:(1)多光谱输入信息丰富但难以一致利用,由于波段冗余和不匹配;(2)CLIP风格的文本编码器限制了语义表达能力并削弱了细粒度对齐。我们提出了SATtxt,一种具有光谱意识的VLFM,它在推理时仅使用RGB输入,同时保留了训练期间学习到的光谱线索。我们的框架分为两个阶段。首先,光谱表示蒸馏将光谱先验从冻结的多光谱教师模型转移到RGB学生模型,通过轻量级的投影器。其次,具有指令增强的大语言模型的光谱接地对齐将蒸馏的视觉空间与一个表达力强的LLM嵌入空间连接起来。在EuroSAT、BigEarthNet和ForestNet上,SATtxt在零样本分类上平均提高了4.2%,检索提高了5.9%,线性探测提高了2.7%,显示出通往地球观测中光谱感知视觉-语言学习的有效路径。项目页面:https://ikhado.github.io/sattxt/
英文摘要:
Vision-language foundation models (VLFMs) promise zero-shot and retrieval understanding for Earth observation. While operational satellite systems often lack full multi-spectral coverage, making RGB-only inference highly desirable for scalable deployment, the adoption of VLFMs for satellite imagery remains hindered by two factors: (1) multi-spectral inputs are informative but difficult to exploit consistently due to band redundancy and misalignment; and (2) CLIP-style text encoders limit semantic expressiveness and weaken fine-grained alignment. We present SATtxt, a spectrum-aware VLFM that operates with RGB inputs only at inference while retaining spectral cues learned during training. Our framework comprises two stages. First, Spectral Representation Distillation transfers spectral priors from a frozen multi-spectral teacher to an RGB student via a lightweight projector. Second, Spectrally Grounded Alignment with Instruction-Augmented LLMs bridges the distilled visual space and an expressive LLM embedding space. Across EuroSAT, BigEarthNet, and ForestNet, SATtxt improves zero-shot classification on average by 4.2%, retrieval by 5.9%, and linear probing by 2.7% over baselines, showing an efficient path toward spectrum-aware vision-language learning for Earth observation. Project page: https://ikhado.github.io/sattxt/
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。