In many intelligent systems, a network of agents collaboratively perceives the environment for better and more efficient situation awareness. As these agents often have limited resources, it could be greatly beneficial to identify the content overlapping among camera views from different agents and leverage it for reducing the processing, transmission and storage of redundant/unimportant video frames. This paper presents a consensus-based distributed multi-agent video fast-forwarding framework, named DMVF, that fast-forwards multi-view video streams collaboratively and adaptively. In our framework, each camera view is addressed by a reinforcement learning based fast-forwarding agent, which periodically chooses from multiple strategies to selectively process video frames and transmits the selected frames at adjustable paces. During every adaptation period, each agent communicates with a number of neighboring agents, evaluates the importance of the selected frames from itself and those from its neighbors, refines such evaluation together with other agents via a system-wide consensus algorithm, and uses such evaluation to decide their strategy for the next period. Compared with approaches in the literature on a real-world surveillance video dataset VideoWeb, our method significantly improves the coverage of important frames and also reduces the number of frames processed in the system.
Northwestern University（西北大学）；University of California, Riverside（加州大学河滨分校）
For many applications with limited computation, communication, storage and energy resources, there is an imperative need of computer vision methods that could select an informative subset of the input video for efficient processing at or near real time. In the literature, there are two relevant groups of approaches: generating a trailer for a video or fast-forwarding while watching/processing the video. The first group is supported by video summarization techniques, which require processing of the entire video to select an important subset for showing to users. In the second group, current fas…
Universidade Federal de Minas Gerais (UFMG)（米纳斯吉拉斯联邦大学（UFMG））；University of California Berkeley（加州大学伯克利分校）
The remarkable technological advance in well-equipped wearable devices is pushing an increasing production of long first-person videos. However, since most of these videos have long and tedious parts, they are forgotten or never seen. Despite a large number of techniques proposed to fast-forward these videos by highlighting relevant moments, most of them are image based only. Most of these techniques disregard other relevant sensors present in the current devices such as high-definition microphones. In this work, we propose a new approach to fast-forward videos using psychoacoustic metrics ext…
Universidade Federal de Minas Gerais (UFMG)（米纳斯吉拉斯联邦大学（UFMG））；Lancaster University（兰卡斯特大学）
The rapid increase in the amount of published visual data and the limited time of users bring the demand for processing untrimmed videos to produce shorter versions that convey the same information. Despite the remarkable progress that has been made by summarization methods, most of them can only select a few frames or skims, which creates visual gaps and breaks the video context. In this paper, we present a novel methodology based on a reinforcement learning formulation to accelerate instructional videos. Our approach can adaptively select frames that are not relevant to convey the informatio…
The emergence of low-cost high-quality personal wearable cameras combined with the increasing storage capacity of video-sharing websites have evoked a growing interest in first-person videos, since most videos are composed of long-running unedited streams which are usually tedious and unpleasant to watch. State-of-the-art semantic fast-forward methods currently face the challenge of providing an adequate balance between smoothness in visual flow and the emphasis on the relevant parts. In this work, we present the Multi-Importance Fast-Forward (MIFF), a fully automatic methodology to fast-forwa…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Visual surveillance systems have become one of the largest data sources of Big Visual Data in real world. However, existing systems for video analysis still lack the ability to handle the problems of scalability, expansibility and error-prone, though great advances have been achieved in a number of visual recognition tasks and surveillance applications, e.g., pedestrian/vehicle detection, people/vehicle counting. Moreover, few algorithms explore the specific values/characteristics in large-scale surveillance videos. To address these problems in large-scale video analysis, we develop a scalable…
While egocentric cameras like GoPro are gaining popularity, the videos they capture are long, boring, and difficult to watch from start to end. Fast forwarding (i.e. frame sampling) is a natural choice for faster video browsing. However, this accentuates the shake caused by natural head motion, making the fast forwarded video useless. We propose EgoSampling, an adaptive frame sampling that gives more stable fast forwarded videos. Adaptive frame sampling is formulated as energy minimization, whose optimal solution can be found in polynomial time. In addition, egocentric video taken while walkin…
The Hebrew University of Jerusalem（耶路撒冷希伯来大学）；IIIT Delhi（德里印度理工学院）
The possibility of sharing one's point of view makes use of wearable cameras compelling. These videos are often long, boring and coupled with extreme shake, as the camera is worn on a moving person. Fast forwarding (i.e. frame sampling) is a natural choice for quick video browsing. However, this accentuates the shake caused by natural head motion in an egocentric video, making the fast forwarded video useless. We propose EgoSampling, an adaptive frame sampling that gives stable, fast forwarded, hyperlapse videos. Adaptive frame sampling is formulated as an energy minimization problem, whose op…
BRAC University（BRAC大学）；University of Delaware（特拉华大学）
Real-time video surveillance, through CCTV camera systems has become essential for ensuring public safety which is a priority today. Although CCTV cameras help a lot in increasing security, these systems require constant human interaction and monitoring. To eradicate this issue, intelligent surveillance systems can be built using deep learning video classification techniques that can help us automate surveillance systems to detect violence as it happens. In this research, we explore deep learning video classification techniques to detect violence as they are happening. Traditional image classi…
Hunan University（湖南大学）；National Supercomputing Center in Changsha（长沙国家超级计算中心）；Xiangtan University（湘潭大学）；State University of New York, New Paltz（纽约州立大学新帕尔茨分校）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Tsinghua University（清华大学）
In this paper, we propose a Distributed Intelligent Video Surveillance (DIVS) system using Deep Learning (DL) algorithms and deploy it in an edge computing environment. We establish a multi-layer edge computing architecture and a distributed DL training model for the DIVS system. The DIVS system can migrate computing workloads from the network center to network edges to reduce huge network communication overhead and provide low-latency and accurate video analysis solutions. We implement the proposed DIVS system and address the problems of parallel training, model synchronization, and workload …
This paper presents a multi-view distributed video coding framework for independent camera encoding and centralized decoding. Spatio-temporal-view concealment methods are developed that exploit the interleaved nature of the employed hybrid KEY/Wyner-Ziv frames for block-wise generation of the side information (SI). We study a number of view concealment methods and develop a joint approach that exploits all available correlation for forming the side information. We apply a diversity technique for fusing multiple such predictions thereby achieving more reliable results. We additionally introduce…
College of Mathematics and Informatics, South China Agricultural University（华南农业大学数学与信息学院）；School of Computer Science and Technology, Guangdong University of Technology（广东工业大学计算机科学与技术学院）；Advanced Digital Sciences Center, Illinois at Singapore Pte.Ltd.（伊利诺伊先进数字科学中心（新加坡））
Traditional dehazing techniques, as a well studied topic in image processing, are now widely used to eliminate the haze effects from individual images. However, even the state-of-the-art dehazing algorithms may not provide sufficient support to video analytics, as a crucial pre-processing step for video-based decision making systems (e.g., robot navigation), due to the limitations of these algorithms on poor result coherence and low processing efficiency. This paper presents a new framework, particularly designed for video dehazing, to output coherent results in real time, with two novel techn…
Indian Institute of Technology, Kharagpur（印度理工学院卡拉格普尔分校）；Azure Software Systems Ltd.（Azure软件系统有限公司）
In this paper, we propose a new feedback-channel-free Distributed Video Coding (DVC) algorithm using Local Rank Transform (LRT). The encoder computes LRT by considering selected neighborhood pixels of Wyner-Ziv frame. The ranks from the modified LRT are merged, and their positions are entropy coded and sent to the decoder. In addition, means of each block of Wyner-Ziv frame are also transmitted to assist motion estimation. Using these measurements, the decoder generates side information (SI) by implementing motion estimation and compensation in LRT domain. An iterative algorithm is executed on…
We present TDNet, a temporally distributed network designed for fast and accurate video semantic segmentation. We observe that features extracted from a certain high-level layer of a deep CNN can be approximated by composing features extracted from several shallower sub-networks. Leveraging the inherent temporal continuity in videos, we distribute these sub-networks over sequential frames. Therefore, at each time step, we only need to perform a lightweight computation to extract a sub-features group from a single sub-network. The full features used for segmentation are then recomposed by appli…
MIT（麻省理工学院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Deep video recognition is more computationally expensive than image recognition, especially on large-scale datasets like Kinetics [1]. Therefore, training scalability is essential to handle a large amount of videos. In this paper, we study the factors that impact the training scalability of video networks. We recognize three bottlenecks, including data loading (data movement from disk to GPU), communication (data movement over networking), and computation FLOPs. We propose three design guidelines to improve the scalability: (1) fewer FLOPs and hardware-friendly operator to increase the computa…
Training a Distributed Acoustic Sensing Traffic Monitoring Network With Video Inputs
School of Physics and Astronomy, Tel-Aviv University（物理与天文学院，特拉维夫大学）；School of Electrical Engineering, Tel-Aviv University（电气工程学院，特拉维夫大学）；Porter School of the Environment and Earth Sciences, Tel-Aviv University（环境与地球科学波特学院，特拉维夫大学）
Shandong University（山东大学）；University of Queensland（昆士兰大学）；Shandong Jianzhu University（山东建筑大学）
Despite the great success achieved, existing video moment retrieval (VMR) methods are developed under the assumption that data are centralizedly stored. However, in real-world applications, due to the inherent nature of data generation and privacy concerns, data are often distributed on different silos, bringing huge challenges to effective large-scale training. In this work, we try to overcome above limitation by leveraging the recent success of federated learning. As the first that is explored in VMR field, the new task is defined as video moment retrieval with distributed data. Then, a nove…
University of Surrey（萨里大学）；The National Archives（国家档案馆）；Open Data Institute (ODI)（开放数据研究所（ODI））
We present ARCHANGEL; a novel distributed ledger based system for assuring the long-term integrity of digital video archives. First, we describe a novel deep network architecture for computing compact temporal content hashes (TCHs) from audio-visual streams with durations of minutes or hours. Our TCHs are sensitive to accidental or malicious content modification (tampering) but invariant to the codec used to encode the video. This is necessary due to the curatorial requirement for archives to format shift video over time to ensure future accessibility. Second, we describe how the TCHs (and the…
A Survey on Video Analytics in Cloud-Edge-Terminal Collaborative Systems
Fudan University（复旦大学）；Duke Kunshan University（昆山杜克大学）；University of Toronto（多伦多大学）；Imperial College London（伦敦帝国学院）；University of California, Davis（加利福尼亚大学戴维斯分校）；SMBU；University of Ottawa（渥太华大学）
