ChA-MAEViT：统一通道感知掩码自编码器与多通道视觉Transformer以改进跨通道学习
ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning
AI总结：
提出ChA-MAEViT，通过动态通道-补丁掩码、记忆令牌、混合令牌融合和通道感知解码器，增强多通道成像中的跨通道学习，在多个数据集上显著优于现有方法。
摘要：
先前使用掩码自编码器（MAEs）的工作通常基于图像在不同通道间存在显著冗余的假设，采用随机补丁掩码，从而可以利用跨通道相关性重建被掩码的内容。然而，这一假设在多通道成像（MCI）中并不成立，因为不同通道可能提供互补信息且特征重叠极少。因此，这些MAEs主要从补丁重建中学习单个通道内的局部结构，未能充分利用跨通道交互，限制了其在MCI中的有效性。在本文中，我们提出了ChA-MAEViT，一种基于MAE的方法，通过四种关键策略增强跨MCI通道的特征学习：（1）动态通道-补丁掩码，迫使模型在重建掩码补丁之外还重建缺失通道，从而增强跨通道依赖性并提高对变化通道配置的鲁棒性；（2）记忆令牌，作为长期记忆辅助以促进跨通道信息共享，应对重建结构多样通道的挑战；（3）混合令牌融合模块，将细粒度补丁令牌与全局类别令牌合并以捕获更丰富的表示；（4）通道感知解码器，一种轻量级解码器，利用通道令牌有效重建图像补丁。在卫星和显微镜数据集CHAMMI、JUMP-CP和So2Sat上的实验表明，ChA-MAEViT显著优于最先进的MCI-ViTs，性能提升3.0%-21.5%，凸显了跨通道交互在MCI中的重要性。我们的代码公开于https://github.com/chaudatascience/cha_mae_vit。
英文摘要：
Prior work using Masked Autoencoders (MAEs) typically relies on random patch masking based on the assumption that images have significant redundancies across different channels, allowing for the reconstruction of masked content using cross-channel correlations. However, this assumption does not hold in Multi-Channel Imaging (MCI), where channels may provide complementary information with minimal feature overlap. Thus, these MAEs primarily learn local structures within individual channels from patch reconstruction, failing to fully leverage cross-channel interactions and limiting their MCI effectiveness. In this paper, we present ChA-MAEViT, an MAE-based method that enhances feature learning across MCI channels via four key strategies: (1) dynamic channel-patch masking, which compels the model to reconstruct missing channels in addition to masked patches, thereby enhancing cross-channel dependencies and improving robustness to varying channel configurations; (2) memory tokens, which serve as long-term memory aids to promote information sharing across channels, addressing the challenges of reconstructing structurally diverse channels; (3) hybrid token fusion module, which merges fine-grained patch tokens with a global class token to capture richer representations; and (4) Channel-Aware Decoder, a lightweight decoder utilizes channel tokens to effectively reconstruct image patches. Experiments on satellite and microscopy datasets, CHAMMI, JUMP-CP, and So2Sat, show that ChA-MAEViT significantly outperforms state-of-the-art MCI-ViTs by 3.0-21.5%, highlighting the importance of cross-channel interactions in MCI. Our code is publicly available at https://github.com/chaudatascience/cha_mae_vit.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
