arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.04470cs.CV
收藏

夜间彩色-热红外语义分割的测试时自适应方法

Test-Time Adaptation for Nighttime Color-Thermal Semantic Segmentation

HKUST(GZ)(香港科技大学(广州));University of Agder(阿格德尔大学);HKUST(香港科技大学)

Yexin Liu, Weiming Zhang, Guoyang Zhao, Jinjing Zhu, Athanasios Vasilakos, Lin Wang

AI总结:

本文提出Night-TTA测试时自适应框架,通过成像异质性细化、类别感知细化和协同学习方案,在无需白天源数据的情况下提升夜间RGB-热红外语义分割性能,mIoU提升13.07%。

摘要:

在夜间等不利视觉条件下进行场景理解的能力,推动了RGB-热红外(RGB-T)语义分割的积极研究。然而,该任务本质上受到两个关键问题的制约:1)RGB图像的昼夜差异大于热红外图像的昼夜差异;2)夜间RGB图像在各类别上的性能并不始终高于或低于热红外图像。我们提出首个测试时自适应(TTA)框架,称为Night-TTA,以在自适应过程中不访问源(白天)数据的情况下解决夜间RGBT语义分割的上述问题。我们的方法包含三个关键技术部分。首先,由于一种模态(如RGB)比另一种模态(如热红外)遭受更大的域差异,成像异质性细化(Imaging Heterogeneity Refinement,IHR)在RGB分支和热红外分支的基础上引入交互分支,以防止跨模态差异和性能退化。然后,引入类别感知细化(Class Aware Refinement,CAR),基于三个分支的像素级分布聚合获得可靠的集成logits。此外,我们还为TTA框架设计了一种特定的学习方案,使集成logits和三个学生logits能够在Night-TTA的测试阶段协同学习,以提高预测质量。大量实验表明,我们的方法取得了最先进(SoTA)的性能,mIoU提升了13.07%。

英文摘要:

The ability to scene understanding in adverse visual conditions, e.g., nighttime, has sparked active research for RGB-Thermal (RGB-T) semantic segmentation. However, it is essentially hampered by two critical problems: 1) the day-night gap of RGB images is larger than that of thermal images, and 2) the class-wise performance of RGB images at night is not consistently higher or lower than that of thermal images. we propose the first test-time adaptation (TTA) framework, dubbed Night-TTA, to address the problems for nighttime RGBT semantic segmentation without access to the source (daytime) data during adaptation. Our method enjoys three key technical parts. Firstly, as one modality (e.g., RGB) suffers from a larger domain gap than that of the other (e.g., thermal), Imaging Heterogeneity Refinement (IHR) employs an interaction branch on the basis of RGB and thermal branches to prevent cross-modal discrepancy and performance degradation. Then, Class Aware Refinement (CAR) is introduced to obtain reliable ensemble logits based on pixel-level distribution aggregation of the three branches. In addition, we also design a specific learning scheme for our TTA framework, which enables the ensemble logits and three student logits to collaboratively learn to improve the quality of predictions during the testing phase of our Night TTA. Extensive experiments show that our method achieves state-of-the-art (SoTA) performance with a 13.07% boost in mIoU.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑