SpikeMba：用于时间视频定位的多模态脉冲显著性曼巴
SpikeMba: Multi-Modal Spiking Saliency Mamba for Temporal Video Grounding
AI总结：
SpikeMba结合脉冲神经网络与状态空间模型，通过脉冲显著性检测器生成提议集并利用相关槽与上下文推理器，解决时间视频定位中的置信度偏差和长期依赖问题，显著提升多模态关系捕捉性能。
摘要：
时间视频定位（TVG）是视频内容理解中的一项关键任务，要求视频内容与自然语言指令之间精确对齐。尽管已有显著进展，现有方法在处理对显著物体的置信度偏差以及捕捉视频序列中的长期依赖关系方面仍面临挑战。为解决这些问题，我们提出了SpikeMba：一种用于时间视频定位的多模态脉冲显著性曼巴。我们的方法将脉冲神经网络（SNNs）与状态空间模型（SSMs）相结合，利用它们在处理任务不同方面的独特优势。具体来说，我们使用SNNs开发了一个脉冲显著性检测器，用于生成提议集。当输入信号超过预定义阈值时，检测器会发出脉冲信号，从而产生动态且二进制的显著性提议集。为了增强模型保留和推断上下文信息的能力，我们引入了相关槽（relevant slots），这些槽是可学习的张量，编码先验知识。这些槽与上下文时刻推理器协同工作，在保留上下文信息和动态探索语义相关性之间保持平衡。SSMs促进了选择性信息传播，解决了视频内容中长期依赖的挑战。通过结合SNNs进行提议生成和SSMs进行有效的上下文推理，SpikeMba解决了置信度偏差和长期依赖问题，从而显著增强了细粒度多模态关系捕捉。我们的实验证明了SpikeMba的有效性，它在主流基准测试中持续优于最先进的方法。
英文摘要：
Temporal video grounding (TVG) is a critical task in video content understanding, requiring precise alignment between video content and natural language instructions. Despite significant advancements, existing methods face challenges in managing confidence bias towards salient objects and capturing long-term dependencies in video sequences. To address these issues, we introduce SpikeMba: a multi-modal spiking saliency mamba for temporal video grounding. Our approach integrates Spiking Neural Networks (SNNs) with state space models (SSMs) to leverage their unique advantages in handling different aspects of the task. Specifically, we use SNNs to develop a spiking saliency detector that generates the proposal set. The detector emits spike signals when the input signal exceeds a predefined threshold, resulting in a dynamic and binary saliency proposal set. To enhance the model's capability to retain and infer contextual information, we introduce relevant slots which learnable tensors that encode prior knowledge. These slots work with the contextual moment reasoner to maintain a balance between preserving contextual information and exploring semantic relevance dynamically. The SSMs facilitate selective information propagation, addressing the challenge of long-term dependency in video content. By combining SNNs for proposal generation and SSMs for effective contextual reasoning, SpikeMba addresses confidence bias and long-term dependencies, thereby significantly enhancing fine-grained multimodal relationship capture. Our experiments demonstrate the effectiveness of SpikeMba, which consistently outperforms state-of-the-art methods across mainstream benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
