具有长期记忆的鲁棒自体-外体对应
Robust Ego-Exo Correspondence with Long-Term Memory
AI总结：
本文提出基于SAM 2的新型EEC框架，通过双记忆架构和自适应特征路由模块提升长期记忆能力，实现更鲁棒的自体-外体对应。
摘要：
建立自体视图和外体视图之间的物体级对应对于智能助手提供精确直观的视觉指导至关重要。然而，该任务面临极端视角变化、遮挡和小物体存在的诸多挑战。现有方法通常借鉴视频物体分割模型的解决方案，但仍存在上述问题。最近，Segment Anything Model 2 (SAM 2) 展示了强大的泛化能力和出色的视频物体分割性能。然而，当简单应用于自体-外体对应（EEC）任务时，SAM 2由于无效的自体-外体特征融合和有限的长期记忆容量，尤其是在长视频中，会遇到严重困难。为解决这些问题，我们提出了一种基于SAM 2的新型EEC框架，通过引入双记忆架构和受Mixture-of-Experts（MoE）启发的自适应特征路由模块。与SAM 2相比，我们的方法具有（i）一个Memory-View MoE模块，该模块包含双分支路由机制，用于自适应地为每个专家特征分配通道和空间维度上的贡献权重，以及（ii）一个双记忆银行系统，采用简单但有效的压缩策略，以保留关键的长期信息并消除冗余。在具有挑战性的EgoExo4D基准测试中，我们的方法，称为LM-EEC，实现了新的最先进的结果，并显著优于现有方法和SAM 2基线，展示了其在多样化场景中的强大泛化能力。我们的代码和模型可在https://github.com/juneyeeHu/LM-EEC获取。
英文摘要：
Establishing object-level correspondence between egocentric and exocentric views is essential for intelligent assistants to deliver precise and intuitive visual guidance. However, this task faces numerous challenges, including extreme viewpoint variations, occlusions, and the presence of small objects. Existing approaches usually borrow solutions from video object segmentation models, but still suffer from the aforementioned challenges. Recently, the Segment Anything Model 2 (SAM 2) has shown strong generalization capabilities and excellent performance in video object segmentation. Yet, when simply applied to the ego-exo correspondence (EEC) task, SAM 2 encounters severe difficulties due to ineffective ego-exo feature fusion and limited long-term memory capacity, especially for long videos. Addressing these problems, we propose a novel EEC framework based on SAM 2 with long-term memories by presenting a dual-memory architecture and an adaptive feature routing module inspired by Mixture-of-Experts (MoE). Compared to SAM 2, our approach features (i) a Memory-View MoE module which consists of a dual-branch routing mechanism to adaptively assign contribution weights to each expert feature along both channel and spatial dimensions, and (ii) a dual-memory bank system with a simple yet effective compression strategy to retain critical long-term information while eliminating redundancy. In the extensive experiments on the challenging EgoExo4D benchmark, our method, dubbed LM-EEC, achieves new state-of-the-art results and significantly outperforms existing methods and the SAM 2 baseline, showcasing its strong generalization across diverse scenarios. Our code and model are available at https://github.com/juneyeeHu/LM-EEC.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
