一种使用语言对齐特征进行多光谱场景表征的视觉-语言框架
A Vision-Language Framework for Multispectral Scene Representation Using Language-Grounded Features
AI总结：
针对遥感复杂场景表征难题，提出视觉-语言框架Spectral LLaVA，结合多光谱数据与对齐技术，在BigEarthNet v2数据集上验证了其在场景描述与分类上的显著提升效果。
摘要：
遥感领域的场景理解在为各类土地利用区域、沿海区域等复杂环境生成准确表征时常常面临挑战，这些环境中还可能存在积雪、云层或雾霾。为解决这一问题，我们提出了名为Spectral LLaVA的视觉-语言框架，该框架将多光谱数据与视觉-语言对齐技术相结合，以提升场景表征与描述能力。我们基于Sentinel-2的BigEarthNet v2数据集，建立了基于RGB的场景描述基线，并进一步证明引入多光谱信息可带来显著性能提升。该框架优化了一个轻量级线性投影层以实现对齐，同时冻结SpectralGPT的视觉骨干网络。我们的实验涵盖了使用线性探测的场景分类任务，以及联合执行场景分类与描述生成的语言建模任务。实验结果凸显了Spectral LLaVA生成详细准确描述的能力，尤其适用于仅靠RGB数据不足以完成任务的场景，同时它还通过将SpectralGPT特征优化为具有语义意义的表征，提升了分类性能。
英文摘要：
Scene understanding in remote sensing often faces challenges in generating accurate representations for complex environments such as various land use areas or coastal regions, which may also include snow, clouds, or haze. To address this, we present a vision-language framework named Spectral LLaVA, which integrates multispectral data with vision-language alignment techniques to enhance scene representation and description. Using the BigEarthNet v2 dataset from Sentinel-2, we establish a baseline with RGB-based scene descriptions and further demonstrate substantial improvements through the incorporation of multispectral information. Our framework optimizes a lightweight linear projection layer for alignment while keeping the vision backbone of SpectralGPT frozen. Our experiments encompass scene classification using linear probing and language modeling for jointly performing scene classification and description generation. Our results highlight Spectral LLaVA's ability to produce detailed and accurate descriptions, particularly for scenarios where RGB data alone proves inadequate, while also enhancing classification performance by refining SpectralGPT features into semantically meaningful representations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
