通过关键令牌的加权适配分割任意事件
Segment Any Events via Weighted Adaptation of Pivotal Tokens
- Xidian University（西安电子科技大学）
- City University of Hong Kong（香港城市大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本文提出多尺度特征蒸馏方法，通过加权适配关键令牌嵌入，将分割任意模型适配于事件数据，实现稳健的通用目标分割。
AI中文摘要：
本文深入探讨了将分割任意模型（SAMs）适配于事件数据整合的细微挑战，其总体目标是在以事件为中心的领域内实现稳健且通用的目标分割。该工作的核心关键问题之一是对源自事件中心的嵌入进行精确对齐和校准，使其与来自RGB图像的嵌入和谐一致。利用大量配对事件与RGB图像的数据集，我们提出利用并外推预训练SAM框架内蕴含的深层知识。作为实现这一目标的基础，我们引入了一种多尺度特征蒸馏方法。该方法严格优化了事件数据令牌嵌入与其RGB图像对应物之间的对齐，从而保持并增强了整体架构的鲁棒性。考虑到中间层令牌嵌入对高层嵌入具有显著重要性，我们的策略侧重于精确校准关键令牌嵌入。这种有针对性的校准旨在有效管理源自事件和图像域的高层嵌入中的差异。在不同数据集上进行的大量实验证明了所提出的蒸馏方法的有效性。代码见http://github.com/happychenpipi/EventSAM。
英文摘要：
In this paper, we delve into the nuanced challenge of tailoring the Segment Anything Models (SAMs) for integration with event data, with the overarching objective of attaining robust and universal object segmentation within the event-centric domain. One pivotal issue at the heart of this endeavor is the precise alignment and calibration of embeddings derived from event-centric data such that they harmoniously coincide with those originating from RGB imagery. Capitalizing on the vast repositories of datasets with paired events and RGB images, our proposition is to harness and extrapolate the profound knowledge encapsulated within the pre-trained SAM framework. As a cornerstone to achieving this, we introduce a multi-scale feature distillation methodology. This methodology rigorously optimizes the alignment of token embeddings originating from event data with their RGB image counterparts, thereby preserving and enhancing the robustness of the overall architecture. Considering the distinct significance that token embeddings from intermediate layers hold for higher-level embeddings, our strategy is centered on accurately calibrating the pivotal token embeddings. This targeted calibration is aimed at effectively managing the discrepancies in high-level embeddings originating from both the event and image domains. Extensive experiments on different datasets demonstrate the effectiveness of the proposed distillation method. Code in http://github.com/happychenpipi/EventSAM.
