混合深度学习框架中的多模态线索建模
Modeling Multimodal Clues in a Hybrid Deep Learning Framework for Video Classification
摘要：
视频本质上是多模态的。本文研究了如何充分利用丰富的多模态线索来改进视频分类的问题。我们引入了一个混合深度学习框架，该框架集成了来自多种模态的有用线索，包括静态空间外观信息、短时间窗口内的运动模式、音频信息以及长距离时间动态。更具体地说，我们利用三个卷积神经网络（CNN）操作的外观，运动和音频信号，以提取其相应的功能。然后，我们采用一个特征融合网络，以获得一个统一的表示，旨在捕捉特征之间的关系。此外，为了利用视频中的长距离时间动态特性，我们应用了两个长短期记忆网络，并将提取的外观和运动特征作为输入。最后，我们还建议利用视频语义之间的上下文关系来改进预测分数。混合深度学习框架能够利用一组全面的多模态特征进行视频分类。通过一组广泛的实验，我们证明了（1）以显式递归方式对序列进行建模的LSTM网络与CNN模型高度互补;（2）通过对特征关系进行建模来产生融合表示的特征融合网络优于其他融合策略;（3）视频类的语义上下文可以帮助进一步改进预测以提高性能。两个具有挑战性的基准测试，UCF-101和哥伦比亚消费者视频（CCV）的实验结果，提供了强有力的定量证据表明，我们的框架取得了可喜的成果：$93.1\%$的UCF-101和$84.5\%$的CCV，优于竞争的方法具有明显的利润。
英文摘要：
Videos are inherently multimodal. This paper studies the problem of how to fully exploit the abundant multimodal clues for improved video categorization. We introduce a hybrid deep learning framework that integrates useful clues from multiple modalities, including static spatial appearance information, motion patterns within a short time window, audio information as well as long-range temporal dynamics. More specifically, we utilize three Convolutional Neural Networks (CNNs) operating on appearance, motion and audio signals to extract their corresponding features. We then employ a feature fusion network to derive a unified representation with an aim to capture the relationships among features. Furthermore, to exploit the long-range temporal dynamics in videos, we apply two Long Short Term Memory networks with extracted appearance and motion features as inputs. Finally, we also propose to refine the prediction scores by leveraging contextual relationships among video semantics. The hybrid deep learning framework is able to exploit a comprehensive set of multimodal features for video classification. Through an extensive set of experiments, we demonstrate that (1) LSTM networks which model sequences in an explicitly recurrent manner are highly complementary with CNN models; (2) the feature fusion network which produces a fused representation through modeling feature relationships outperforms alternative fusion strategies; (3) the semantic context of video classes can help further refine the predictions for improved performance. Experimental results on two challenging benchmarks, the UCF-101 and the Columbia Consumer Videos (CCV), provide strong quantitative evidence that our framework achieves promising results: $93.1\%$ on the UCF-101 and $84.5\%$ on the CCV, outperforming competing methods with clear margins.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
