Understanding temporal information and how the visual world changes over time is a fundamental ability of intelligent systems. In video understanding, temporal information is at the core of many current challenges, including compression, efficient inference, motion estimation or summarization. However, in current video datasets it has been observed that action classes can often be recognized without any temporal information from a single frame of video. As a result, both benchmarking and training in these datasets may give an unintentional advantage to models with strong image understanding capabilities, as opposed to those with strong temporal understanding. In this paper we address this problem head on by identifying action classes where temporal information is actually necessary to recognize them and call these "temporal classes". Selecting temporal classes using a computational method would bias the process. Instead, we propose a methodology based on a simple and effective human annotation experiment. We remove just the temporal information by shuffling frames in time and measure if the action can still be recognized. Classes that cannot be recognized when frames are not in order are included in the temporal Dataset. We observe that this set is statistically different from other static classes, and that performance in it correlates with a network's ability to capture temporal information. Thus we use it as a benchmark on current popular networks, which reveals a series of interesting facts. We also explore the effect of training on the temporal dataset, and observe that this leads to better generalization in unseen classes, demonstrating the need for more temporal data. We hope that the proposed dataset of temporal categories will help guide future research in temporal modeling for better video understanding.
Learn to cycle: Time-consistent feature discovery for action recognition
Utrecht University(乌得勒支大学)
在时间变化上进行概括是视频中有效动作识别的先决条件。尽管深度神经网络取得了重大进展,但关注与动作的整体表现相关的短期辨别性运动仍然是一个挑战。我们通过允许一些灵活性来发现相关的时空特征来应对这一挑战。我们介绍了挤压和递归时间门(SRTG),一种方法,有利于输入类似的激活与潜在的时间变化。我们用一个新的CNN块来实现这个想法,该块使用LSTM来封装特征动态,并结合一个时间门来评估发现的动态和建模特征的一致性。当使用SRTG块时,我们表现出一致的改进,GFLOPs的数量只有最小的增加。在Kinetics-700上,我们的表现与当前最先进的模型相当,并且在HACS,Moments in Time,UCF-101和HMDB-51上表现优于这些模型。
DORi: Discovering Object Relationship for Moment Localization of a Natural-Language Query in Video
Australian National University(澳大利亚国立大学);Australian Centre for Robotic Vision (ACRV)(澳大利亚机器人视觉中心(ACRV));Graduate School of Engineering, The University of Tokyo(东京大学工学研究科);A*AI, A*STAR Singapore(A*AI,新加坡A*STAR)
Agent-Environment Network for Temporal Action Proposal Generation
University of Science, VNU-HCM(越南国家大学胡志明市分校科学大学);Vietnam National University, Ho Chi Minh City(越南国家大学胡志明市分校);University of Arkansas(阿肯色大学);National Institute of Informatics(国立信息学研究所)
Discovering Sounding Objects by Audio Queries for Audio Visual Segmentation
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机学院);Alibaba Group(阿里巴巴集团);Institute of Artificial Intelligence, Beihang University(北京航空航天大学人工智能研究院);Hangzhou Innovation Institute, Beihang University(北京航空航天大学杭州创新研究院)
What and When to Look?: Temporal Span Proposal Network for Video Relation Detection
Korea Advanced Institute of Science and Technology(韩国科学技术院);Gwangju Institute of Science and Technology(光州科学技术院);Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室)
Video Activity Localisation with Uncertainties in Temporal Boundary
Queen Mary University of London(伦敦玛丽女王大学);Vision Semantics Limited(视觉语义有限公司);Adobe Research(Adobe研究院);Peking University(北京大学);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院)