StreamMeCo: Long-Term Agent Memory Compression for Efficient Streaming Video Understanding
Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）；Shanghai AI Laboratory（上海人工智能实验室）；Hong Kong University of Science and Technology（香港科学与技术大学）；MemTensor (Shanghai) Technology Co., Ltd.（MemTensor（上海）技术有限公司）
Vision agent memory has shown remarkable effectiveness in streaming video understanding. However, storing such memory for videos incurs substantial memory overhead, leading to high costs in both storage and computation. To address this issue, we propose StreamMeCo, an efficient Stream Agent Memory Compression framework. Specifically, based on the connectivity of the memory graph, StreamMeCo introduces edge-free minmax sampling for the isolated nodes and an edge-aware weight pruning for connected nodes, evicting the redundant memory nodes while maintaining the accuracy. In addition, we introduce a time-decay memory retrieval mechanism to further eliminate the performance degradation caused by memory compression. Extensive experiments on three challenging benchmark datasets (M3-Bench-robot, M3-Bench-web and Video-MME-Long) demonstrate that under 70% memory graph compression, StreamMeCo achieves a 1.87* speedup in memory retrieval while delivering an average accuracy improvement of 1.0%. Our code is available at https://github.com/Celina-love-sweet/StreamMeCo.
Accelerating Streaming Video Large Language Models via Hierarchical Token Compression
EPIC Lab, Shanghai Jiao Tong University（上海交通大学EPIC实验室）；Sichuan University（四川大学）；Huazhong University of Science and Technology（华中科技大学）；Sun Yat-sen University（中山大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding
Institute of Trustworthy Embodied AI（可信具身人工智能研究所）；Fudan University（复旦大学）；Shanghai Innovation Institute（上海创新研究院）；Shanghai Key Laboratory of Multimodal Embodied AI（上海多模态具身人工智能重点实验室）；University of Maryland, College Park（马里兰大学 College Park 分校）
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China（脑启发式智能感知与认知联合实验室，中国科学技术大学）；University of Missouri, Kansas City（密苏里大学，堪萨斯城）
4DGCPro: Efficient Hierarchical 4D Gaussian Compression for Progressive Volumetric Video Streaming
Cooperative Medianet Innovation Center, Shanghai Jiaotong University（合作中位网创新中心，上海交通大学）；Department of Electronics, Shanghai Jiaotong University（电子工程系，上海交通大学）；Shanghai AI Lab（上海人工智能实验室）；Cloud platform department, E-surfing Vision Technology Co., Ltd.（云平台部门，E-surfing Vision Technology有限公司）；School of Information Science and Technology, ShanghaiTech University（信息科学与技术学院，上海科技大学）
FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understanding
College of Future Information Technology, Fudan University, Shanghai, China（复旦大学未来信息科技学院）；Shanghai Innovation Institute, Shanghai, China（上海创新研究院）；Shanghai Artificial Intelligence Laboratory, Shanghai, China（上海人工智能实验室）；The Chinese University of Hong Kong, Hong Kong, China（香港中文大学）
Shenzhen International Graduate School Tsinghua University（清华大学深圳国际研究生院）；Institute of Artificial Intelligence Xiamen University（厦门大学人工智能研究院）；China Star Optoelectronics Technology（中国星电科技）；TCL Corporate Research (HK) Co Ltd.（TCL香港企业研发中心）；Department of Electrical and Electronic Engineering The University of Hong Kong（香港大学电子与电气工程系）
Waseda University（早稻田大学）；Waseda Research Institute for Science and Engineering（早稻田大学科学与工程研究院）
Learned image compression allows achieving state-of-the-art accuracy and compression ratios, but their relatively slow runtime performance limits their usage. While previous attempts on optimizing learned image codecs focused more on the neural model and entropy coding, we present an alternative method to improving the runtime performance of various learned image compression models. We introduce multi-threaded pipelining and an optimized memory model to enable GPU and CPU workloads asynchronous execution, fully taking advantage of computational resources. Our architecture alone already produce…
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Institute of Artificial Intelligence, Xiamen University（厦门大学人工智能学院）；China Star Optoelectronics Technology（中国星电科技）；TCL Corporate Research (HK) Co., Ltd.（TCL香港企业研究有限公司）；Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子与电气工程系）
StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression
Peking University（北京大学）；Nanjing University of Science and Technology（南京理工大学）；University of Science and Technology of China（中国科学技术大学）；Zhejiang University（浙江大学）；Huawei（华为）；Tongji University（同济大学）
University of California, Merced（加州大学默塞德分校）；NVIDIA（英伟达）
We study domain-specific video streaming. Specifically, we target a streaming setting where the videos to be streamed from a server to a client are all in the same domain and they have to be compressed to a small size for low-latency transmission. Several popular video streaming services, such as the video game streaming services of GeForce Now and Twitch, fall in this category. While conventional video compression standards such as H.264 are commonly used for this task, we hypothesize that one can leverage the property that the videos are all in the same domain to achieve better video quality…
School of Electrical Engineering and Automation, Hefei University of Technology, and Engineering Technology Research Center of Industrial Automation Anhui Province, Hefei, Anhui 230009, China（合肥工业大学电气与自动化学院，安徽省工业自动化工程技术研究中心）；The University of Electro-Communications, Tokyo, Japan（东京电讯大学）；School of Computer Science and Technology, University of Science and Technology of China, Hefei, China（中国科学技术大学计算机科学与技术学院）；Department of Electrical and Computer Engineering, Aarhus University, Aarhus, Denmark（哥本哈根大学电子与计算机工程系）；Department of Computer Science and Electrical Engineering, University of Missouri–Kansas City, Kansas, USA（密苏里大学- Kansas 城市分校计算机科学与电气工程系）；School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China（合肥工业大学计算机科学与信息工程学院）
We present an explicit-grid based method for efficiently reconstructing streaming radiance fields for novel view synthesis of real world dynamic scenes. Instead of training a single model that combines all the frames, we formulate the dynamic modeling problem with an incremental learning paradigm in which per-frame model difference is trained to complement the adaption of a base model on the current frame. By exploiting the simple yet effective tuning strategy with narrow bands, the proposed method realizes a feasible framework for handling video sequences on-the-fly with high training efficie…
CAGS: Color-Adaptive Volumetric Video Streaming with Dynamic 3D Gaussian Splatting
Simon Fraser University（西蒙弗雷泽大学）；Jiangxing Intelligence Inc.（江行智能有限公司）；McGill University（麦吉尔大学）；Nankai University（南开大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Fuzhou University（福州大学）；Southeast University（东南大学）
OASIS: On-Demand Hierarchical Event Memory for Streaming Video Reasoning
Sun Yat-sen University（中山大学）；OPPO AI Center（OPPO人工智能中心）；Shenzhen Loop Area Institute（深圳河套学院）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东大数据分析与处理重点实验室）
Most video super-resolution methods focus on restoring high-resolution video frames from low-resolution videos without taking into account compression. However, most videos on the web or mobile devices are compressed, and the compression can be severe when the bandwidth is limited. In this paper, we propose a new compression-informed video super-resolution model to restore high-resolution content without introducing artifacts caused by compression. The proposed model consists of three modules for video super-resolution: bi-directional recurrent warping, detail-preserving flow estimation, and L…
Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video
ShanghaiTech University（上海科技大学）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；School of Computing and Data Science, The University of Hong Kong（香港大学计算科学与数据科学学院）
Recent years have witnessed the dramatic growth of Internet video traffic, where the video bitstreams are often compressed and delivered in low quality to fit the streamer's uplink bandwidth. To alleviate the quality degradation, it comes the rise of Neural-enhanced Video Streaming (NVS), which shows great prospects for recovering low-quality videos by mostly deploying neural super-resolution (SR) on the media server. Despite its benefit, we reveal that current mainstream works with SR enhancement have not achieved the desired rate-distortion trade-off between bitrate saving and quality restor…
Qualcomm AI Research, Qualcomm Technologies Netherlands B.V.（高通人工智能研究院，高通技术荷兰有限公司）；Qualcomm AI Research, Qualcomm Technologies, Inc.（高通人工智能研究院，高通技术公司）；University of Oxford（牛津大学）
Recent advances in deep generative modeling have enabled efficient modeling of high dimensional data distributions and opened up a new horizon for solving data compression problems. Specifically, autoencoder based learned image or video compression solutions are emerging as strong competitors to traditional approaches. In this work, We propose a new network architecture, based on common and well studied components, for learned video compression operating in low latency mode. Our method yields state of the art MS-SSIM/rate performance on the high-resolution UVG dataset, among both learned video…
Recent works in spatiotemporal radiance fields can produce photorealistic free-viewpoint videos. However, they are inherently unsuitable for interactive streaming scenarios (e.g. video conferencing, telepresence) because have an inevitable lag even if the training is instantaneous. This is because these approaches consume videos and thus have to buffer chunks of frames (often seconds) before processing. In this work, we take a step towards interactive streaming via a frame-by-frame approach naturally free of lag. Conventional wisdom believes that per-frame NeRFs are impractical due to prohibit…
University of California San Diego（加州大学圣地亚哥分校）；KDDI Research, Inc.（KDDI研究所）
Video compression plays a crucial role in video streaming and classification systems by maximizing the end-user quality of experience (QoE) at a given bandwidth budget. In this paper, we conduct the first systematic study for adversarial attacks on deep learning-based video compression and downstream classification systems. Our attack framework, dubbed RoVISQ, manipulates the Rate-Distortion ($\textit{R}$-$\textit{D}$) relationship of a video compression model to achieve one or both of the following goals: (1) increasing the network bandwidth, (2) degrading the video quality for end-users. We …
Graduate School of Science and Engineering, Hosei University（法政大学理工学研究生院）
COVID-19 leads to the high demand for remote interactive systems ever seen. One of the key elements of these systems is video streaming, which requires a very high network bandwidth due to its specific real-time demand, especially with high-resolution video. Existing video compression methods are struggling in the trade-off between video quality and the speed requirement. Addressed that the background information rarely changes in most remote meeting cases, we introduce a Region-Of-Interests (ROI) based video compression framework (named RCLC) that leverages the cutting-edge learning-based and…
