Thermo-VL：扩展视觉语言模型以适应热红外感知
Thermo-VL: Extending Vision-Language Models to Thermal Infrared Perception
AI总结：
本文提出Thermo-VL，一种基于热红外感知的视觉语言模型，通过引入可训练的热编码器和文本引导的双注意力融合模块，提升了低光照条件下的多光谱融合能力，并在热红外和RGB+热红外推理任务中取得显著成果。
摘要：
视觉语言模型（VLMs）在低光照条件下往往表现不佳，因为它们的视觉基础主要学习自RGB图像，而热红外图像在可见线索退化时能保留互补的场景结构。我们提出了Thermo-VL，一种波长感知的VLM，它在冻结的Molmo-7B主干上添加了可训练的热编码器和文本引导的双注意力融合模块。给定对齐的RGB标记、热标记和提示嵌入，融合模块将热特征条件化为语言和RGB上下文，然后将门控残差注入冻结的RGB流中，使热证据能够被纳入而不破坏Molmo预训练的RGB-语言接口。我们使用标准的语言建模目标以及辅助对齐和正则化损失来训练模型，这些损失提高了跨模态基础并减少了对RGB的依赖。我们还引入了一个像素对齐的RGB-热指令微调数据集和Thermo-VL-Bench，一个手动筛选的RGB-热VQA基准，用于低光照和跨光谱推理。实验表明，在具有挑战性的热红外和RGB+热红外推理任务中取得了显著的提升，突显了基于提示的多光谱融合的价值。我们的数据集和代码可在：https://thusharakart.github.io/Thermo-VL 公开获取。
英文摘要：
Vision-language models (VLMs) often fail under low illumination because their visual grounding is learned predominantly from RGB imagery, whereas thermal infrared preserves complementary scene structure when visible cues degrade. We present Thermo-VL, a wavelength-aware VLM that augments a frozen Molmo-7B backbone with a trainable thermal encoder and a text-guided dual-attention fusion module. Given aligned RGB tokens, thermal tokens, and prompt embeddings, the fusion module conditions thermal features on both language and RGB context, then injects a gated residual into the frozen RGB stream so thermal evidence can be incorporated without disrupting Molmo's pretrained RGB-language interface. We train the model with the standard language-modeling objective together with auxiliary alignment and regularization losses that improve cross-modal grounding and reduce over-reliance on RGB. We also introduce a pixel-aligned RGB-thermal instruction-tuning dataset and Thermo-VL-Bench, a manually screened RGB-thermal VQA benchmark for low-light and cross-spectrum reasoning. Experiments show strong gains on challenging thermal-only and RGB+thermal reasoning tasks, highlighting the value of prompt-conditioned multispectral fusion. Our dataset and code are publicly available at: https://thusharakart.github.io/Thermo-VL
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
