PiLaMIM：通过整合像素与潜在掩码图像建模以获得更丰富的视觉表示
PiLaMIM: Toward Richer Visual Representations by Integrating Pixel and Latent Masked Image Modeling
AI总结：
本文提出PiLaMIM统一框架，通过单编码器与像素、潜在双解码器并引入CLS token，融合低层细节与高层语义，实验表明其在多数情况下优于MAE、I-JEPA和BootMAE。
摘要：
在掩码图像建模（MIM）中，主要存在两种方法：Pixel MIM和Latent MIM，它们分别使用不同的重建目标，即原始像素和潜在表示。Pixel MIM倾向于捕获颜色和纹理等低层视觉细节，而Latent MIM则关注对象的高层语义。然而，每种方法的这些独特优势可能导致其在依赖特定层级视觉特征的任务中表现欠佳。为解决这一局限，我们提出PiLaMIM，这是一个统一框架，结合Pixel MIM和Latent MIM，以整合二者的互补优势。我们的方法使用单个编码器以及两个不同的解码器：一个用于预测像素值，另一个用于预测潜在表示，从而确保同时捕获高层和低层视觉特征。我们进一步将CLS token整合到重建过程中，以聚合全局上下文，使模型能够捕获更多语义信息。大量实验表明，PiLaMIM在大多数情况下优于MAE、I-JEPA和BootMAE等关键基线，证明了其在提取更丰富视觉表示方面的有效性。
英文摘要：
In Masked Image Modeling (MIM), two primary methods exist: Pixel MIM and Latent MIM, each utilizing different reconstruction targets, raw pixels and latent representations, respectively. Pixel MIM tends to capture low-level visual details such as color and texture, while Latent MIM focuses on high-level semantics of an object. However, these distinct strengths of each method can lead to suboptimal performance in tasks that rely on a particular level of visual features. To address this limitation, we propose PiLaMIM, a unified framework that combines Pixel MIM and Latent MIM to integrate their complementary strengths. Our method uses a single encoder along with two distinct decoders: one for predicting pixel values and another for latent representations, ensuring the capture of both high-level and low-level visual features. We further integrate the CLS token into the reconstruction process to aggregate global context, enabling the model to capture more semantic information. Extensive experiments demonstrate that PiLaMIM outperforms key baselines such as MAE, I-JEPA and BootMAE in most cases, proving its effectiveness in extracting richer visual representations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
