arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2207.01398cs.CVeess.IV
收藏

视频动作识别模型的大规模鲁棒性分析

Large-scale Robustness Analysis of Video Action Recognition Models

CRCV(计算机视觉研究中心);University of Central Florida(中佛罗里达大学);IIT Kanpur(印度理工学院坎普尔分校);Microsoft Research(微软研究院)

Madeline Chantry Schiappa, Naman Biyani, Prudvi Kamtam, Shruti Vyas, Hamid Palangi, Vibhav Vineet, Yogesh Rawat

AI总结:

本文对六种视频动作识别模型在90种真实世界分布偏移扰动下的鲁棒性进行了大规模分析,提出了四个扰动基准数据集和一个真实分布偏移数据集,发现基于Transformer的模型比CNN模型更鲁棒,且时间信息的重要性因数据集而异。

摘要:

近年来,我们在视频动作识别领域取得了巨大进展。现有多种基于卷积神经网络(CNN)的模型以及一些近期基于transformer的方法,它们在现有基准上提供了顶尖性能。在这项工作中,我们对这些现有的视频动作识别模型进行了大规模的鲁棒性分析。我们关注的是针对真实世界分布偏移扰动的鲁棒性,而非对抗性扰动。我们提出了四个不同的基准数据集,HMDB51-P、UCF101-P、Kinetics400-P和SSv2-P,来执行此分析。我们研究了六种最先进的动作识别模型对90种不同扰动的鲁棒性。研究揭示了一些有趣的发现:1)与基于CNN的模型相比,基于transformer的模型始终具有更强的鲁棒性;2)预训练对基于Transformer模型鲁棒性的提升大于基于CNN的模型;3)除SSv2外,所有研究模型对所有数据集的时间扰动均具有鲁棒性;这表明时间信息对动作识别的重要性因数据集和活动而异。接下来,我们研究了数据增强在模型鲁棒性中的作用,并提出了一个包含真实分布偏移的真实世界数据集UCF101-DS,以进一步验证部分发现。我们相信这项研究将作为未来鲁棒视频动作识别研究的基准。

英文摘要:

We have seen a great progress in video action recognition in recent years. There are several models based on convolutional neural network (CNN) and some recent transformer based approaches which provide top performance on existing benchmarks. In this work, we perform a large-scale robustness analysis of these existing models for video action recognition. We focus on robustness against real-world distribution shift perturbations instead of adversarial perturbations. We propose four different benchmark datasets, HMDB51-P, UCF101-P, Kinetics400-P, and SSv2-P to perform this analysis. We study robustness of six state-of-the-art action recognition models against 90 different perturbations. The study reveals some interesting findings, 1) transformer based models are consistently more robust compared to CNN based models, 2) Pretraining improves robustness for Transformer based models more than CNN based models, and 3) All of the studied models are robust to temporal perturbations for all datasets but SSv2; suggesting the importance of temporal information for action recognition varies based on the dataset and activities. Next, we study the role of augmentations in model robustness and present a real-world dataset, UCF101-DS, which contains realistic distribution shifts, to further validate some of these findings. We believe this study will serve as a benchmark for future research in robust video action recognition.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑