arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1907.08340cs.CVcs.LG
收藏

只有时间能告诉我们:发现时态建模的时态数据

Only Time Can Tell: Discovering Temporal Data for Temporal Modeling

University of Edinburgh(爱丁堡大学);Facebook AI(脸书人工智能研究院)

Laura Sevilla-Lara, Shengxin Zha, Zhicheng Yan, Vedanuj Goswami, Matt Feiszli, Lorenzo Torresani

摘要:

理解时间信息以及视觉世界如何随时间变化是智能系统的基本能力。在视频理解中,时间信息是当前许多挑战的核心,包括压缩、高效推理、运动估计或摘要。然而,在当前的视频数据集中,已经观察到动作类通常可以在没有来自单个视频帧的任何时间信息的情况下被识别。因此,这些数据集中的基准测试和训练可能会给具有强大图像理解能力的模型带来无意的优势,而不是那些具有强大时间理解能力的模型。在本文中,我们解决这个问题的头上确定的动作类的时间信息实际上是必要的,以识别它们,并呼吁这些“时间类”。使用计算方法选择时态类会使过程产生偏差。相反,我们提出了一种基于简单有效的人类注释实验的方法。我们通过在时间上重排帧来去除时间信息,并测量动作是否仍然可以被识别。当帧不按顺序排列时无法识别的类包含在时态数据集中。我们观察到,这个集合在统计上与其他静态类不同,并且它的性能与网络捕获时间信息的能力相关。因此,我们将其作为当前流行网络的基准,这揭示了一系列有趣的事实。我们还探索了训练对时态数据集的影响,并观察到这导致了在看不见的类中更好的泛化,这表明需要更多的时态数据。我们希望所提出的时间类别的数据集将有助于指导未来的时间建模更好的视频理解的研究。

英文摘要:

Understanding temporal information and how the visual world changes over time is a fundamental ability of intelligent systems. In video understanding, temporal information is at the core of many current challenges, including compression, efficient inference, motion estimation or summarization. However, in current video datasets it has been observed that action classes can often be recognized without any temporal information from a single frame of video. As a result, both benchmarking and training in these datasets may give an unintentional advantage to models with strong image understanding capabilities, as opposed to those with strong temporal understanding. In this paper we address this problem head on by identifying action classes where temporal information is actually necessary to recognize them and call these "temporal classes". Selecting temporal classes using a computational method would bias the process. Instead, we propose a methodology based on a simple and effective human annotation experiment. We remove just the temporal information by shuffling frames in time and measure if the action can still be recognized. Classes that cannot be recognized when frames are not in order are included in the temporal Dataset. We observe that this set is statistically different from other static classes, and that performance in it correlates with a network's ability to capture temporal information. Thus we use it as a benchmark on current popular networks, which reveals a series of interesting facts. We also explore the effect of training on the temporal dataset, and observe that this leads to better generalization in unseen classes, demonstrating the need for more temporal data. We hope that the proposed dataset of temporal categories will help guide future research in temporal modeling for better video understanding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑