SalFoM：基于视频基础模型的动态显著性预测
SalFoM: Dynamic Saliency Prediction with Video Foundation Models
AI总结：
本文提出 SalFoM，首次基于视频基础模型构建编码器-解码器视频显著性预测架构，利用 UMT 与异构时空解码器融合多视角局部和全局信息，并在 DHF1K、Hollywood-2、UCF-Sports 上超越现有方法。
摘要：
近期，视频显著性预测（VSP）的进展已展现出与人类视觉系统相比颇具前景的性能，而模拟人类视觉系统正是 VSP 的首要目标。然而，当前最先进的模型采用在有限数据上训练的时空 transformer，这限制了其对下游任务的泛化适应能力。视觉基础模型所带来的优势为改进 VSP 流程提供了一种潜在解决方案。但是，将图像基础模型适配到视频领域，在对场景动态进行建模以及捕获时间信息方面存在重大挑战。为应对这些挑战，并作为设计基于视频基础模型的 VSP 模型的首次尝试，我们提出了 SalFoM，这是一种新颖的编码器-解码器视频 transformer 架构。我们的模型采用 UnMasked Teacher（UMT）作为特征提取器，并提出了一个异构解码器；该解码器以局部性感知的时空 transformer 为特色，从多个视角整合局部与全局时空信息，以生成最终的显著性图。我们在具有挑战性的 VSP 基准数据集 DHF1K、Hollywood-2 和 UCF-Sports 上进行的定性与定量实验，证明了所提模型相较于最先进方法的优越性。
英文摘要：
Recent advancements in video saliency prediction (VSP) have shown promising performance compared to the human visual system, whose emulation is the primary goal of VSP. However, current state-of-the-art models employ spatio-temporal transformers trained on limited amounts of data, hindering generalizability adaptation to downstream tasks. The benefits of vision foundation models present a potential solution to improve the VSP process. However, adapting image foundation models to the video domain presents significant challenges in modeling scene dynamics and capturing temporal information. To address these challenges, and as the first initiative to design a VSP model based on video foundation models, we introduce SalFoM, a novel encoder-decoder video transformer architecture. Our model employs UnMasked Teacher (UMT) as feature extractor and presents a heterogeneous decoder which features a locality-aware spatio-temporal transformer and integrates local and global spatio-temporal information from various perspectives to produce the final saliency map. Our qualitative and quantitative experiments on the challenging VSP benchmark datasets of DHF1K, Hollywood-2 and UCF-Sports demonstrate the superiority of our proposed model in comparison with the state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
