arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.12975cs.CV
收藏

基于单张图像与事件的实例级运动目标分割

Instance-Level Moving Object Segmentation from a Single Image with Events

Northwestern Polytechnical University(西北工业大学);National University of Singapore(新加坡国立大学);Peking University(北京大学)

Zhexiong Wan, Bin Fan, Le Hui, Yuchao Dai, Gim Hee Lee

AI总结:

针对单模态方法在运动目标分割中空间纹理与时间运动线索难以兼顾的问题,提出首个融合单张图像纹理与事件运动信息的实例级分割框架,通过跨模态注意力增强等技术提升性能,为事件基运动研究提供新方向。

摘要:

运动目标分割在理解包含多个运动目标的动态场景中起着至关重要的作用,其难点在于同时考虑空间纹理结构和时间运动线索。现有基于视频帧的方法由于精确的基于图像的运动建模复杂,难以区分目标的像素位移是由相机运动还是目标运动引起的。近期进展利用新型事件相机的运动敏感性来弥补传统图像运动建模能力不足的问题,但由于事件缺乏密集纹理结构,反而导致分割像素级目标掩码面临挑战。为解决单模态设置带来的这两个局限性,我们提出首个融合互补纹理和运动线索的实例级运动目标分割框架。我们的模型分别结合隐式跨模态掩码注意力增强、显式对比特征学习和流引导运动增强,以利用单张图像的密集纹理信息和事件的丰富运动信息。通过利用增强的纹理和运动特征,我们将掩码分割与运动分类分离,以处理数量变化的独立运动目标。通过在多个数据集上的广泛评估,以及不同输入设置的消融实验和所提框架的实时效率分析,我们认为首次尝试融合图像和事件数据以实现实际部署,可为未来基于事件的运动相关研究提供新见解。包含模型训练和预训练权重的源代码已发布于https://npucvr.github.io/EvInsMOS

英文摘要:

Moving object segmentation plays a crucial role in understanding dynamic scenes involving multiple moving objects, while the difficulties lie in taking into account both spatial texture structures and temporal motion cues. Existing methods based on video frames encounter difficulties in distinguishing whether pixel displacements of an object are caused by camera motion or object motion due to the complexities of accurate image-based motion modeling. Recent advances exploit the motion sensitivity of novel event cameras to counter conventional images' inadequate motion modeling capabilities, but instead lead to challenges in segmenting pixel-level object masks due to the lack of dense texture structures in events. To address these two limitations imposed by unimodal settings, we propose the first instance-level moving object segmentation framework that integrates complementary texture and motion cues. Our model incorporates implicit cross-modal masked attention augmentation, explicit contrastive feature learning, and flow-guided motion enhancement to exploit dense texture information from a single image and rich motion information from events, respectively. By leveraging the augmented texture and motion features, we separate mask segmentation from motion classification to handle varying numbers of independently moving objects. Through extensive evaluations on multiple datasets, as well as ablation experiments with different input settings and real-time efficiency analysis of the proposed framework, we believe that our first attempt to incorporate image and event data for practical deployment can provide new insights for future work in event-based motion related works. The source code with model training and pre-trained weights is released at https://npucvr.github.io/EvInsMOS

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑