arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.09969cs.CVcs.AIcs.CL
收藏

视频分类的迁移学习:Video Swin Transformer在多领域的应用

Transfer-learning for video classification: Video Swin Transformer on multiple domains

INESC-ID;Instituto Superior Técnico(高等技术学院);Universidade de Lisboa(里斯本大学)

Daniel A. P. Oliveira, David Martins de Matos

AI总结:

本文研究Video Swin Transformer在视频分类中的域外泛化能力,通过从Kinetics-400迁移学习,发现当目标类别与训练集同类时表现优异,不同类时准确率显著下降。

摘要:

计算机视觉社区在图像和视频任务中已经看到了从基于卷积到纯 Transformer 架构的转变。为这些任务从零开始训练 Transformer 通常需要大量数据和计算资源。Video Swin Transformer (VST) 是一种为视频分类开发的纯 Transformer 模型,在多个数据集上实现了准确率和效率的最先进结果。在本文中,我们旨在了解 VST 是否具有足够的泛化能力以用于域外设置。我们使用从 Kinetics-400 进行迁移学习的方法,研究了 VST 在两个大规模数据集(即 FCVID 和 Something-Something)上的性能,该方法所需的内存比从头开始训练少约 4 倍。然后我们分解结果,以了解 VST 在哪里失败最多,以及在哪些场景下迁移学习方法是可行的。我们的实验表明,在 FCVID 上无需重新训练整个模型即可达到 85% 的 top-1 准确率,这等于该数据集的最先进水平,而在 Something-Something 上的准确率为 21%。实验还表明,随着视频持续时间的增加,VST 的性能平均会下降,这似乎是模型设计选择的结果。从结果中我们得出结论,当目标类别与用于训练模型的类别属于同一类型时,VST 的泛化能力足以在不重新训练的情况下对域外视频进行分类。当我们执行从 Kinetics-400 到 FCVID 的迁移学习时,我们观察到了这种效果,其中大多数数据集主要针对对象。另一方面,如果类别不属于同一类型,那么迁移学习方法后的准确率预计会很差。当我们执行从 Kinetics-400(类别主要代表对象)到 Something-Something(类别主要代表动作)的迁移学习时,我们观察到了这种效果。

英文摘要:

The computer vision community has seen a shift from convolutional-based to pure transformer architectures for both image and video tasks. Training a transformer from zero for these tasks usually requires a lot of data and computational resources. Video Swin Transformer (VST) is a pure-transformer model developed for video classification which achieves state-of-the-art results in accuracy and efficiency on several datasets. In this paper, we aim to understand if VST generalizes well enough to be used in an out-of-domain setting. We study the performance of VST on two large-scale datasets, namely FCVID and Something-Something using a transfer learning approach from Kinetics-400, which requires around 4x less memory than training from scratch. We then break down the results to understand where VST fails the most and in which scenarios the transfer-learning approach is viable. Our experiments show an 85\% top-1 accuracy on FCVID without retraining the whole model which is equal to the state-of-the-art for the dataset and a 21\% accuracy on Something-Something. The experiments also suggest that the performance of the VST decreases on average when the video duration increases which seems to be a consequence of a design choice of the model. From the results, we conclude that VST generalizes well enough to classify out-of-domain videos without retraining when the target classes are from the same type as the classes used to train the model. We observed this effect when we performed transfer-learning from Kinetics-400 to FCVID, where most datasets target mostly objects. On the other hand, if the classes are not from the same type, then the accuracy after the transfer-learning approach is expected to be poor. We observed this effect when we performed transfer-learning from Kinetics-400, where the classes represent mostly objects, to Something-Something, where the classes represent mostly actions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑