This paper proposes a low latency neural network architecture for event-based dense prediction tasks. Conventional architectures encode entire scene contents at a fixed rate regardless of their temporal characteristics. Instead, the proposed network encodes contents at a proper temporal scale depending on its movement speed. We achieve this by constructing temporal hierarchy using stacked latent memories that operate at different rates. Given low latency event steams, the multi-level memories gradually extract dynamic to static scene contents by propagating information from the fast to the slow memory modules. The architecture not only reduces the redundancy of conventional architectures but also exploits long-term dependencies. Furthermore, an attention-based event representation efficiently encodes sparse event streams into the memory cells. We conduct extensive evaluations on three event-based dense prediction tasks, where the proposed approach outperforms the existing methods on accuracy and latency, while demonstrating effective event and image fusion capabilities. The code is available at https://hamarh.github.io/hmnet/
Hierarchical Event Memory for Accurate and Low-latency Online Video Temporal Grounding
Wangxuan Institute of Computer Technology, Peking University(北京大学王宣计算机技术研究所);State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室);Central Media Technology Institute, Huawei(华为中央媒体技术研究所)
VTM-Nav: Harnessing Cross-Episode Experience for Object-Goal Navigation with Hierarchical Visual-Topological Memory
MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所模式识别国家重点实验室);School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学交叉科学学院);Tsinghua University(清华大学);University of Chinese Academy of Sciences(中国科学院大学)
CLIMB: Centroid-Based Hierarchical Memory for Online Continual Self-Supervised Learning
Universite Claude Bernard Lyon 1, CNRS, INSA Lyon, LIRIS, UMR 5205(里昂第一大学, 法国国家科学研究中心, 法国国立应用科学学院, 里昂信息与系统研究实验室, UMR 5205);INSA Lyon, CNRS, Universite Claude Bernard Lyon 1, LIRIS, UMR 5205(法国国立应用科学学院, 法国国家科学研究中心, 里昂第一大学, 里昂信息与系统研究实验室, UMR 5205);Universite Claude Bernard Lyon 1 / Univ. Rennes, Inria, CNRS LIRIS UMR 5205 / IRISA UMR 6074(里昂第一大学 / 雷恩大学, 法国国家信息与自动化研究所, 法国国家科学研究中心 里昂信息与系统研究实验室 UMR 5205 / 雷恩计算机科学研究所 UMR 6074)