Embodied perception refers to the ability of an autonomous agent to perceive its environment so that it can (re)act. The responsiveness of the agent is largely governed by latency of its processing pipeline. While past work has studied the algorithmic trade-off between latency and accuracy, there has not been a clear metric to compare different methods along the Pareto optimal latency-accuracy curve. We point out a discrepancy between standard offline evaluation and real-time applications: by the time an algorithm finishes processing a particular frame, the surrounding world has changed. To these ends, we present an approach that coherently integrates latency and accuracy into a single metric for real-time online perception, which we refer to as "streaming accuracy". The key insight behind this metric is to jointly evaluate the output of the entire perception stack at every time instant, forcing the stack to consider the amount of streaming data that should be ignored while computation is occurring. More broadly, building upon this metric, we introduce a meta-benchmark that systematically converts any single-frame task into a streaming perception task. We focus on the illustrative tasks of object detection and instance segmentation in urban video streams, and contribute a novel dataset with high-quality and temporally-dense annotations. Our proposed solutions and their empirical analysis demonstrate a number of surprising conclusions: (1) there exists an optimal "sweet spot" that maximizes streaming accuracy along the Pareto optimal latency-accuracy curve, (2) asynchronous tracking and future forecasting naturally emerge as internal representations that enable streaming perception, and (3) dynamic scheduling can be used to overcome temporal aliasing, yielding the paradoxical result that latency is sometimes minimized by sitting idle and "doing nothing".
SPARE: Shared Perception and Adaptive Retrieval of Evidence for Streaming Video Understanding
Xiamen University（厦门大学）；Beijing Institute of Technology（北京理工大学）；Fudan University（复旦大学）；Shanghai Jiao Tong University（上海交通大学）；University of Hong Kong（香港大学）；China Telecom（中国电信）
DyRoNet: Dynamic Routing and Low-Rank Adapters for Autonomous Driving Streaming Perception
Institute of Artificial Intelligence, Southwest Jiaotong University（西南交通大学人工智能研究院）；School of Engineering and Technology, University of Washington（华盛顿大学工程与技术学院）；Institute for Intelligent Computing, Alibaba Group（阿里巴巴集团智能计算研究所）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）
MTD: Multi-Timestep Detector for Delayed Streaming Perception
School of Computer and Electronic Information, Guangxi University（广西大学计算机与电子信息学院）；Guangxi Intelligent Digital Services Research Center of Engineering Technology（广西智能数字服务工程技术研究中心）
