Mamba-OTR：一种基于Mamba的未剪辑第一人称视频在线取放检测解决方案
Mamba-OTR: a Mamba-based Solution for Online Take and Release Detection from Untrimmed Egocentric Video
AI总结：
本文提出基于Mamba架构的Mamba-OTR模型，通过focal loss和与评估指标对齐的正则化方案解决未剪辑第一人称视频中取放在线检测的标签不平衡与效率问题，在EPIC-KITCHENS-100上显著优于transformer和普通Mamba基线。
摘要：
本工作解决未剪辑第一人称视频中物体取放（Take and Release，OTR）的在线检测问题。该任务具有挑战性，原因在于严重的标签不平衡、正样本标注在时间上稀疏，以及需要精确的时间预测。此外，方法需要在计算上高效，以便部署于真实世界的在线场景中。为应对这些挑战，我们提出Mamba-OTR，一个基于Mamba架构的模型。Mamba-OTR旨在推理时利用时间递归特性，同时在短视频片段上进行训练。为解决标签不平衡问题，我们的训练流程结合了focal loss和一种新颖的正则化方案，该方案使模型预测与评估指标对齐。在EPIC-KITCHENS-100上的大量实验、与基于transformer方法的比较，以及对不同训练和测试方案的评估，证明了Mamba-OTR在准确性和效率上的优越性。这些发现在评估全长视频或高帧率序列时尤为明显，即使为了计算方便而在短视频片段上训练也是如此。所提出的Mamba-OTR在滑动窗口方式下取得了显著的mp-mAP 45.48，在流式模式下为43.35，而普通transformer为20.32，普通Mamba为25.16，从而为OTR提供了强大的基线。我们将公开发布Mamba-OTR的源代码以支持未来研究。
英文摘要：
This work tackles the problem of Online detection of Take and Release (OTR) of an object in untrimmed egocentric videos. This task is challenging due to severe label imbalance, with temporally sparse positive annotations, and the need for precise temporal predictions. Furthermore, methods need to be computationally efficient in order to be deployed in real-world online settings. To address these challenges, we propose Mamba-OTR, a model based on the Mamba architecture. Mamba-OTR is designed to exploit temporal recurrence during inference while being trained on short video clips. To address label imbalance, our training pipeline incorporates the focal loss and a novel regularization scheme that aligns model predictions with the evaluation metric. Extensive experiments on EPIC-KITCHENS-100, the comparisons with transformer-based approach, and the evaluation of different training and test schemes demonstrate the superiority of Mamba-OTR in both accuracy and efficiency. These finding are particularly evident when evaluating full-length videos or high frame-rate sequences, even when trained on short video snippets for computational convenience. The proposed Mamba-OTR achieves a noteworthy mp-mAP of 45.48 when operating in a sliding-window fashion, and 43.35 in streaming mode, versus the 20.32 of a vanilla transformer and 25.16 of a vanilla Mamba, thus providing a strong baseline for OTR. We will publicly release the source code of Mamba-OTR to support future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
