We study domain-specific video streaming. Specifically, we target a streaming setting where the videos to be streamed from a server to a client are all in the same domain and they have to be compressed to a small size for low-latency transmission. Several popular video streaming services, such as the video game streaming services of GeForce Now and Twitch, fall in this category. While conventional video compression standards such as H.264 are commonly used for this task, we hypothesize that one can leverage the property that the videos are all in the same domain to achieve better video quality. Based on this hypothesis, we propose a novel video compression pipeline. Specifically, we first apply H.264 to compress domain-specific videos. We then train a novel binary autoencoder to encode the leftover domain-specific residual information frame-by-frame into binary representations. These binary representations are then compressed and sent to the client together with the H.264 stream. In our experiments, we show that our pipeline yields consistent gains over standard H.264 compression across several benchmark datasets while using the same channel bandwidth.
StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding
Tsinghua University（清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；OPPO AI Center, OPPO Inc.（OPPO人工智能中心，OPPO公司）
EvoStreaming: Your Offline Video Model Is a Natively Streaming Assistant
EPIC Lab, Shanghai Jiao Tong University（上海交通大学EPIC实验室）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Fudan University（复旦大学）
Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；Institute for Interdisciplinary Information Sciences (IIIS), Tsinghua University（清华大学交叉信息研究院（IIIS））；Nanjing University（南京大学）；University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）
The MORPHOLO C++ extended Library allows to convert a specific stereoscopic snapshot into a Native multi-view image through morphing algorithms taking into account display calibration data for specific slanted lenticular 3D monitors. MORPHOLO can also be implemented for glasses-free live applicatons of 3D video streaming, and for diverse innovative scientific, engineering and 3D video game applications -see http://www.morpholo.it
University of Macau（澳门大学）；vivo AI Lab（vivo人工智能实验室）
With the prevailing of live video streaming, establishing an online pixelation method for privacy-sensitive objects is an urgency. Caused by the inaccurate detection of privacy-sensitive objects, simply migrating the tracking-by-detection structure into the online form will incur problems in target initialization, drifting, and over-pixelation. To cope with the inevitable but impacting detection issue, we propose a novel Privacy-sensitive Objects Pixelation (PsOP) framework for automatic personal privacy filtering during live video streaming. Leveraging pre-trained detection networks, our PsOP…
StreamingCoT: A Dataset for Temporal Dynamics and Multimodal Chain-of-Thought Reasoning in Streaming VideoQA
Henan Institute of Advanced Technology, Zhengzhou University（河南高级技术研究所，郑州大学）；Institute of Automation, CAS（自动化研究所，中国科学院）；UCAS（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）
University of Waterloo（滑铁卢大学）；Shanghai Jiao Tong University（上海交通大学）；Ningbo University（宁波大学）
Quality of experience (QoE) assessment for adaptive video streaming plays a significant role in advanced network management systems. It is especially challenging in case of dynamic adaptive streaming schemes over HTTP (DASH) which has increasingly complex characteristics including additional playback issues. In this paper, we provide a brief overview of adaptive video streaming quality assessment. Upon our review of related works, we analyze and compare different variations of objective QoE assessment models with or without using machine learning techniques for adaptive video streaming. Throug…
Streaming Video Generation with Streaming Force Control
Northeastern University（东北大学）；Impossible Research；University of California, Berkeley（加州大学伯克利分校）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts
Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；State Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；X-LANCE Lab, Shanghai Jiao Tong University（上海交通大学X-LANCE实验室）
Learning Streaming Video Representation via Multitask Training
School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Shanghai Innovation Institute（上海创新研究院）；Fudan University（复旦大学）；Shanghai AI Laboratory（上海人工智能实验室）
SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Kuaishou Technology（快手科技）；Zhengzhou University（郑州大学）；ShanghaiTech University（上海科技大学）；Peng Cheng Laboratory（鹏城实验室）
IBM Almaden Research Center（IBM阿尔马登研究中心）；Arizona State University（亚利桑那州立大学）
Many applications benefit from sampling algorithms where a small number of well chosen samples are used to generalize different properties of a large dataset. In this paper, we use diverse sampling for streaming video summarization. Several emerging applications support streaming video, but existing summarization algorithms need access to the entire video which requires a lot of memory and computational power. We propose a memory efficient and computationally fast, online algorithm that uses competitive learning for diverse sampling. Our algorithm is a generalization of online K-means such tha…
Oklahoma State University（俄克拉荷马州立大学）；University of South Florida（南佛罗里达大学）
Event perception tasks such as recognizing and localizing actions in streaming videos are essential for scaling to real-world application contexts. We tackle the problem of learning actor-centered representations through the notion of continual hierarchical predictive learning to localize actions in streaming videos without the need for training labels and outlines for the objects in the video. We propose a framework driven by the notion of hierarchical predictive learning to construct actor-centered features by attention-based contextualization. The key idea is that predictable features or ob…
University of California San Diego（加州大学圣地亚哥分校）；Korea Electronics Technology Institute（韩国电子技术研究院）；Cornell NYC Tech（康奈尔纽约科技学院）
Despite recent advances in video segmentation, many opportunities remain to improve it using a variety of low and mid-level visual cues. We propose improvements to the leading streaming graph-based hierarchical video segmentation (streamGBH) method based on early and mid level visual processing. The extensive experimental analysis of our approach validates the improvement of hierarchical supervoxel representation by incorporating motion and color with effective filtering. We also pose and illuminate some open questions towards intermediate level video analysis as further extension to streamGBH…
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；JD.COM（京东）
Online Streaming Video Super-Resolution with Convolutional Look-Up Table
Zhejiang University（浙江大学）；Tongji University（同济大学）；University of Science and Technology of China（中国科学技术大学）；Shanghai Jiao Tong University（上海交通大学）；Microsoft Research Asia（微软亚洲研究院）
This paper presents a deep-learning based traffic classification method for identifying multiple streaming video sources at the same time within an encrypted tunnel. The work defines a novel feature inspired by Natural Language Processing (NLP) that allows existing NLP techniques to help the traffic classification. The feature extraction method is described, and a large dataset containing video streaming and web traffic is created to verify its effectiveness. Results are obtained by applying several NLP methods to show that the proposed method performs well on both binary and multilabel traffi…
Recognising human activities from streaming videos poses unique challenges to learning algorithms: predictive models need to be scalable, incrementally trainable, and must remain bounded in size even when the data stream is arbitrarily long. Furthermore, as parameter tuning is problematic in a streaming setting, suitable approaches should be parameterless, and make no assumptions on what class labels may occur in the stream. We present here an approach to the recognition of human actions from streaming data which meets all these requirements by: (1) incrementally learning a model which adaptiv…
