Human mesh recovery (HMR) provides a geometric foundation for understanding human motion, but accurately reconstructing both body motion and fine hand articulation from videos remains challenging, especially in real-world scenes where occlusion and partial visibility are common. Video generation pretraining offers rich spatial and temporal priors for this task, but its focus on visual synthesis leaves a gap between these representations and explicit human geometry. We present DreamHMR, a general framework that adapts video generation priors for complete body-and-hand reconstruction. It follows a global-to-local strategy, first grounding video representations in human geometry and then refining body and hand detail with reliable local evidence. Global Body Representation Transfer trains the video backbone to predict complete human renderings, turning its native prediction task into a means of learning surface correspondence and posed geometry. Local Joint Feature Enhancement combines contextual estimates with reliable local evidence to recover fine body and hand detail, while limiting interference from occluders and background. Experiments on five datasets with four video backbones demonstrate strong reconstruction performance across backbone families. Under a unified evaluation protocol, our best model outperforms the best existing methods on all reported metrics of 3DPW, EMDB-1, and RICH, reducing body MPJPE by 4.2%, 4.7%, and 6.1%, respectively. In more challenging occlusion settings on held-out BEDLAM and EgoBody sequences, DreamHMR reduces overall body and hand MPJPE by 7.0% and 10.0%, respectively. Our code will be made publicly available.
Hierarchical Curriculum Learning for Surgical Video Generation: Focusing on Bilateral Axillo-Breast Approach Robotic Thyroidectomy Datasets
Seoul National University College of Medicine（首尔国立大学医学院）；Seoul National University Hospital（首尔国立大学医院）；YONSEI WITH-YOU Surgery Clinic（延世WITH-YOU外科诊所）；Institute of Convergence Medicine with Innovative Technology, Seoul National University Hospital（首尔国立大学医院创新技术融合医学研究所）
Video Forgery Detection for Surveillance Cameras: A Review
Kurdistan Technical Institute（库尔德斯坦技术学院）；University of Kurdistan Hewlêr（库尔德斯坦赫威尔大学）；University of Raparin（拉帕林大学）；Charmo University（恰尔莫大学）；Tishk International University（蒂什克国际大学）；Maharaja Sriram Chandra Bhanja Deo University（马哈拉贾·斯里拉姆·钱德拉·班贾·德奥大学）
Zhejiang University（浙江大学）；Zhejiang Gongshang University（浙江工商大学）；National University of Singapore（新加坡国立大学）；Shandong Normal Unversity（山东师范大学）；Nanyang Technological University（南洋理工大学）；University of Science and Technology of China（中国科学技术大学）
One compelling application of artificial intelligence is to generate a video of a target person performing arbitrary desired motion (from a source person). While the state-of-the-art methods are able to synthesize a video demonstrating similar broad stroke motion details, they are generally lacking in texture details. A pertinent manifestation appears as distorted face, feet, and hands, and such flaws are very sensitively perceived by human observers. Furthermore, current methods typically employ GANs with a L2 loss to assess the authenticity of the generated videos, inherently requiring a lar…
Predicting future frames in natural video sequences is a new challenge that is receiving increasing attention in the computer vision community. However, existing models suffer from severe loss of temporal information when the predicted sequence is long. Compared to previous methods focusing on generating more realistic contents, this paper extensively studies the importance of sequential order information for video generation. A novel Shuffling sEquence gEneration network (SEE-Net) is proposed that can learn to discriminate unnatural sequential orders by shuffling the video frames and comparin…
VRWKV-Editor: Reducing quadratic complexity in transformer-based video editing
International Artificial Intelligence Center of Morocco University Mohammed VI Polytechnic Rabat, Morocco（摩洛哥国际人工智能中心摩洛哥穆莱·伊沙克·穆莱·阿卜杜勒阿齐兹 polytechnic 大学拉巴特分校）；University Mohammed VI Polytechnic Rabat, Morocco（摩洛哥穆莱·伊沙克·穆莱·阿卜杜勒阿齐兹 polytechnic 大学拉巴特分校）；Sorbonne University, LIP6 - UMR 7606 CNRS, France（索邦大学，LIP6 - UMR 7606 CNRS，法国）
Sequence-Adaptive Video Prediction in Continuous Streams using Diffusion Noise Optimization
University of Bonn（波恩大学）；Toyota Motor Europe（丰田欧洲公司）；University of Wuerzburg（乌尔姆大学）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔人工智能与机器学习研究所）
Consistent and Invariant Generalization Learning for Short-video Misinformation Detection
Zhejiang Normal University（浙江师范大学）；Hong Kong University of Science and Technology（香港理工大学）；Zhejiang University（浙江大学）；Tencent（腾讯）；Soochow University（苏州大学）
针对短视频虚假信息检测中的域泛化难题，本文提出 DOCTOR 模型，通过跨模态插值、插值蒸馏及扩散引导去噪学习一致性与域不变特征。
Borealis AI；Simon Fraser University（西蒙菲莎大学）；University of British Columbia（不列颠哥伦比亚大学）；Vector Institute for AI（Vector人工智能研究所）
Human activity videos involve rich, varied interactions between people and objects. In this paper we develop methods for generating such videos -- making progress toward addressing the important, open problem of video generation in complex scenes. In particular, we introduce the task of generating human-object interaction videos in a zero-shot compositional setting, i.e., generating videos for action-object compositions that are unseen during training, having seen the target action and target object separately. This setting is particularly important for generalization in human activity video g…
Institute of Computing Technology（计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Huawei Technologies（华为技术有限公司）；University of Science and Technology of China（中国科学技术大学）
In recent years, creative content generations like style transfer and neural photo editing have attracted more and more attention. Among these, cartoonization of real-world scenes has promising applications in entertainment and industry. Different from image translations focusing on improving the style effect of generated images, video cartoonization has additional requirements on the temporal consistency. In this paper, we propose a spatially-adaptive semantic alignment framework with perceptual motion consistency for coherent video cartoonization in an unsupervised manner. The semantic align…
Tencent AI Lab（腾讯AI实验室）；Beijing Institute of Technology（北京理工大学）；University of California, Berkeley（加州大学伯克利分校）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
In this work, we build upon existing methods for occlusion-aware 3D pose detection in videos. We implement a two stage architecture that consists of the stacked hourglass network to produce 2D pose predictions, which are then inputted into a temporal convolutional network to produce 3D pose predictions. To facilitate prediction on poses with occluded joints, we introduce an intuitive generalization of the cylinder man model used to generate occlusion labels. We find that the occlusion-aware network is able to achieve a mean-per-joint-position error 5 mm less than our linear baseline model on t…
Visual-only self-supervised learning has achieved significant improvement in video representation learning. Existing related methods encourage models to learn video representations by utilizing contrastive learning or designing specific pretext tasks. However, some models are likely to focus on the background, which is unimportant for learning video representations. To alleviate this problem, we propose a new view called long-range residual frame to obtain more motion-specific information. Based on this, we propose the Motion-Contrastive Perception Network (MCPNet), which consists of two branc…
Université de Montréal（蒙特利尔大学）；Goethe-Universität Frankfurt（法兰克福歌德大学）
The task of the emotion recognition in the wild (EmotiW) Challenge is to assign one of seven emotions to short video clips extracted from Hollywood style movies. The videos depict acted-out emotions under realistic conditions with a large degree of variation in attributes such as pose and illumination, making it worthwhile to explore approaches which consider combinations of features from multiple modalities for label assignment. In this paper we present our approach to learning several specialist models using deep learning techniques, each focusing on one modality. Among these are a convoluti…
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Sportradar（Sportradar公司）；UCLouvain（鲁汶大学）；EPFL（洛桑联邦理工学院）；EVS Broadcast Equipment（EVS广播设备公司）；Baidu Research（百度研究院）；Sharif University of Technology（谢里夫理工大学）；Footovision（Footovision公司）；Zewail City of Science, Technology and Innovation（扎维尔科学技术与创新城）；Universitat de Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；Nagoya University（名古屋大学）；Research Center for Applied Mathematics and Machine Intelligence, Zhejiang Lab（之江实验室应用数学与机器智能研究中心）；AIBrain（AIBrain公司）；OPPO Research Institute（OPPO研究院）；University of the Bundeswehr Munich（慕尼黑联邦国防军大学）；Meituan（美团公司）；Tencent Youtu Lab（腾讯优图实验室）；Amazon Prime Video Sport（亚马逊Prime Video体育部门）；Sogang University（西江大学）；The University of Tokyo（东京大学）；Spiideo（Spiideo公司）；University of Tsukuba（筑波大学）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Normandie Univ, INSA Rouen, LITIS（诺曼底大学、鲁茵国立应用科学学院、LITIS机构）
Learning new skills by observing humans' behaviors is an essential capability of AI. In this work, we leverage instructional videos to study humans' decision-making processes, focusing on learning a model to plan goal-directed actions in real-life videos. In contrast to conventional action recognition, goal-directed actions are based on expectations of their outcomes requiring causal knowledge of potential consequences of actions. Thus, integrating the environment structure with goals is critical for solving this task. Previous works learn a single world model will fail to distinguish various …
There is significant progress in recognizing traditional human activities from videos focusing on highly distinctive actions involving discriminative body movements, body-object and/or human-human interactions. Driver's activities are different since they are executed by the same subject with similar body parts movements, resulting in subtle changes. To address this, we propose a novel framework by exploiting the spatiotemporal attention to model the subtle changes. Our model is named Coarse Temporal Attention Network (CTA-Net), in which coarse temporal branches are introduced in a trainable g…
Motion blur is one of the most common degradation artifacts in dynamic scene photography. This paper reviews the NTIRE 2020 Challenge on Image and Video Deblurring. In this challenge, we present the evaluation results from 3 competition tracks as well as the proposed solutions. Track 1 aims to develop single-image deblurring methods focusing on restoration quality. On Track 2, the image deblurring methods are executed on a mobile platform to find the balance of the running speed and the restoration accuracy. Track 3 targets developing video deblurring methods that exploit the temporal relation…
Beihang University（北京航空航天大学）；Sun Yat-sen University（中山大学）；Pazhou Lab（琶洲实验室）；ByteDance AI Lab（字节跳动人工智能实验室）；University of Sydney（悉尼大学）
In this work, we introduce a novel task - Humancentric Spatio-Temporal Video Grounding (HC-STVG). Unlike the existing referring expression tasks in images or videos, by focusing on humans, HC-STVG aims to localize a spatiotemporal tube of the target person from an untrimmed video based on a given textural description. This task is useful, especially for healthcare and security-related applications, where the surveillance videos can be extremely long but only a specific person during a specific period of time is concerned. HC-STVG is a video grounding task that requires both spatial (where) and…
KAUST（阿卜杜拉国王科技大学）；University of Liège（列日大学）；EVS Broadcast Equipment（EVS广播设备公司）；Sportradar（Sportradar公司）；UCLouvain（鲁汶天主教大学）；EPFL（洛桑联邦理工学院）；Baidu Research（百度研究院）
The SoccerNet 2022 challenges were the second annual video understanding challenges organized by the SoccerNet team. In 2022, the challenges were composed of 6 vision-based tasks: (1) action spotting, focusing on retrieving action timestamps in long untrimmed videos, (2) replay grounding, focusing on retrieving the live moment of an action shown in a replay, (3) pitch localization, focusing on detecting line and goal part elements, (4) camera calibration, dedicated to retrieving the intrinsic and extrinsic camera parameters, (5) player re-identification, focusing on retrieving the same players…
Pixels operate locally. Superpixels have some potential to collect information across many pixels; supervoxels have more potential by implicitly operating across time. In this paper, we explore this well established notion thoroughly analyzing how supervoxels can be used in place of and in conjunction with other means of aggregating information across space-time. Focusing on the problem of strictly unsupervised video object segmentation, we devise a method called supervoxel gerrymandering that links masks of foregroundness and backgroundness via local and non-local consensus measures. We pose …
NAVER AI Lab（NAVER人工智能实验室）；KAIST AI（韩国科学技术院人工智能学院）；SNU AIIS（首尔国立大学人工智能研究所）
Efficient video-language modeling should consider the computational cost because of a large, sometimes intractable, number of video frames. Parametric approaches such as the attention mechanism may not be ideal since its computational cost quadratically increases as the video length increases. Rather, previous studies have relied on offline feature extraction or frame sampling to represent the video efficiently, focusing on cross-modal modeling in short video clips. In this paper, we propose a semi-parametric video-grounded text generation model, SeViT, a novel perspective on scalable video-la…
Learning actions from human demonstration video is promising for intelligent robotic systems. Extracting the exact section and re-observing the extracted video section in detail is important for imitating complex skills because human motions give valuable hints for robots. However, the general video understanding methods focus more on the understanding of the full frame,lacking consideration on extracting accurate sections and aligning them with the human's intent. We propose a Learning-from-Observation framework that splits and understands a video of a human demonstration with verbal instruct…
Sun Yat-sen University（中山大学）；Tencent Youtu Lab（腾讯优图实验室）；Xia Men University（厦门大学）
Self-supervised learning has shown great potentials in improving the video representation ability of deep neural networks by getting supervision from the data itself. However, some of the current methods tend to cheat from the background, i.e., the prediction is highly dependent on the video background instead of the motion, making the model vulnerable to background changes. To mitigate the model reliance towards the background, we propose to remove the background impact by adding the background. That is, given a video, we randomly select a static frame and add it to every other frames to cons…
Bangladesh University of Engineering and Technology (BUET)（孟加拉国工程技术大学）
In the advent of wearable body-cameras, human activity classification from First-Person Videos (FPV) has become a topic of increasing importance for various applications, including in life-logging, law-enforcement, sports, workplace, and healthcare. One of the challenging aspects of FPV is its exposure to potentially sensitive objects within the user's field of view. In this work, we developed a privacy-aware activity classification system focusing on office videos. We utilized a Mask-RCNN with an Inception-ResNet hybrid as a feature extractor for detecting, and then blurring out sensitive obj…
While stochastic video prediction models enable future prediction under uncertainty, they mostly fail to model the complex dynamics of real-world scenes. For example, they cannot provide reliable predictions for scenes with a moving camera and independently moving foreground objects in driving scenarios. The existing methods fail to fully capture the dynamics of the structured world by only focusing on changes in pixels. In this paper, we assume that there is an underlying process creating observations in a video and propose to factorize it into static and dynamic components. We model the stat…
University of Liege (ULiège)（列日大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Sportradar（Sportradar公司）；UCLouvain（鲁汶大学）；EPFL（洛桑联邦理工学院）；EVS Broadcast Equipment（EVS广播设备公司）；Baidu Research（百度研究院）；NASK National Research Institute（NASK国家研究院）；Université Libre de Bruxelles (ULB)（布鲁塞尔自由大学）；Sharif University of Technology（谢里夫理工大学）；Footovision（Footovision公司）；Mobius Labs（Mobius Labs公司）；Institut de Robòtica i Informàtica Industrial, CSIC-UPC（CSIC-UPC机器人与工业自动化研究所）；TAHAKOM（TAHAKOM公司）；AIBrain（AIBrain公司）
Video compression artifact reduction aims to recover high-quality videos from low-quality compressed videos. Most existing approaches use a single neighboring frame or a pair of neighboring frames (preceding and/or following the target frame) for this task. Furthermore, as frames of high quality overall may contain low-quality patches, and high-quality patches may exist in frames of low quality overall, current methods focusing on nearby peak-quality frames (PQFs) may miss high-quality details in low-quality frames. To remedy these shortcomings, in this paper we propose a novel end-to-end deep…
Adversarial robustness of deep neural networks is an extensively studied problem in the literature and various methods have been proposed to defend against adversarial images. However, only a handful of defense methods have been developed for defending against attacked videos. In this paper, we propose a novel Over-and-Under complete restoration network for Defending against adversarial videos (OUDefend). Most restoration networks adopt an encoder-decoder architecture that first shrinks spatial dimension then expands it back. This approach learns undercomplete representations, which have large…
Action recognition is so far mainly focusing on the problem of classification of hand selected preclipped actions and reaching impressive results in this field. But with the performance even ceiling on current datasets, it also appears that the next steps in the field will have to go beyond this fully supervised classification. One way to overcome those problems is to move towards less restricted scenarios. In this context we present a large-scale real-world dataset designed to evaluate learning techniques for human action recognition beyond hand-crafted datasets. To this end we put the proces…
Clustering is a ubiquitous tool in unsupervised learning. Most of the existing self-supervised representation learning methods typically cluster samples based on visually dominant features. While this works well for image-based self-supervision, it often fails for videos, which require understanding motion rather than focusing on background. Using optical flow as complementary information to RGB can alleviate this problem. However, we observe that a naive combination of the two views does not provide meaningful gains. In this paper, we propose a principled way to combine two views. Specificall…
