SPKLIP：通过自然语言对齐尖峰视频流
SPKLIP: Aligning Spike Video Streams with Natural Language
AI总结：
SPKLIP提出了一种专门用于尖峰视频-语言对齐的架构，通过分层尖峰特征提取器和尖峰-文本对比学习实现多尺度时间动态建模，并在稀疏异步输出上实现高效的少样本学习。
摘要：
尖峰相机具有独特的传感能力，但其稀疏、异步输出对语义理解构成挑战，尤其在尖峰视频-语言对齐（Spike-VLA）任务中，传统模型如CLIP因模态不匹配而表现不佳。本文提出SPKLIP，首个专为Spike-VLA设计的架构。SPKLIP采用分层尖峰特征提取器，适应性地建模事件流的多尺度时间动态，并利用尖峰-文本对比学习直接对齐尖峰视频与语言，实现有效的少样本学习。全尖峰视觉编码器变体整合了SNN组件，展示了增强的能量效率。实验表明，SPKLIP在基准尖峰数据集上达到最先进的性能，并在新贡献的现实世界数据集上表现出强大的少样本泛化能力。SPKLIP的能量效率凸显了其在类脑部署中的潜力，推动了基于事件的多模态研究。源代码和数据集可在[被匿名化的链接]获取。
英文摘要：
Spike cameras offer unique sensing capabilities but their sparse, asynchronous output challenges semantic understanding, especially for Spike Video-Language Alignment (Spike-VLA) where models like CLIP underperform due to modality mismatch. We introduce SPKLIP, the first architecture specifically for Spike-VLA. SPKLIP employs a hierarchical spike feature extractor that adaptively models multi-scale temporal dynamics in event streams, and uses spike-text contrastive learning to directly align spike video with language, enabling effective few-shot learning. A full-spiking visual encoder variant, integrating SNN components into our pipeline, demonstrates enhanced energy efficiency. Experiments show state-of-the-art performance on benchmark spike datasets and strong few-shot generalization on a newly contributed real-world dataset. SPKLIP's energy efficiency highlights its potential for neuromorphic deployment, advancing event-based multimodal research. The source code and dataset are available at [link removed for anonymity].
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
