本文提出 KP-RNN 神经网络用于人体舞蹈运动预测，并结合 Take The Lead 数据集与 Everybody Dance Now 流水线验证其有效性，为表演艺术数字化和虚拟人动画提供基线。
摘要：
数字化合成人体运动本质上是一个复杂过程，这会在虚拟现实等应用领域造成障碍。我们提出了一种用于预测人体运动的新方法 KP-RNN，这是一种能够轻松与现有图像处理和生成流水线集成的神经网络。我们利用一个新的表演艺术人体运动数据集 Take The Lead，以及运动生成流水线 Everybody Dance Now 系统，来证明 KP-RNN 运动预测的有效性。我们发现，我们的神经网络能够有效预测人类舞蹈动作，这为未来使用 Take The Lead 数据集的研究提供了基线结果。由于 KP-RNN 能够与 Everybody Dance Now 这样的系统协同工作，我们认为该方法可以启发渲染人体化身动画的新方法。这项工作还通过利用易于使用的神经网络，有助于在数字平台上实现表演艺术的可视化。
英文摘要：
Digitally synthesizing human motion is an inherently complex process, which can create obstacles in application areas such as virtual reality. We offer a new approach for predicting human motion, KP-RNN, a neural network which can integrate easily with existing image processing and generation pipelines. We utilize a new human motion dataset of performance art, Take The Lead, as well as the motion generation pipeline, the Everybody Dance Now system, to demonstrate the effectiveness of KP-RNN's motion predictions. We have found that our neural network can predict human dance movements effectively, which serves as a baseline result for future works using the Take The Lead dataset. Since KP-RNN can work alongside a system such as Everybody Dance Now, we argue that our approach could inspire new methods for rendering human avatar animation. This work also serves to benefit the visualization of performance art in digital platforms by utilizing accessible neural networks.
Recognizing a piece of writing as a poem or prose is usually easy for the majority of people; however, only specialists can determine which meter a poem belongs to. In this paper, we build Recurrent Neural Network (RNN) models that can classify poems according to their meters from plain text. The input text is encoded at the character level and directly fed to the models without feature handcrafting. This is a step forward for machine understanding and synthesis of languages in general, and Arabic language in particular. Among the 16 poem meters of Arabic and the 4 meters of English the networ…
Existing keyframe-based motion synthesis mainly focuses on the generation of cyclic actions or short-term motion, such as walking, running, and transitions between close postures. However, these methods will significantly degrade the naturalness and diversity of the synthesized motion when dealing with complex and impromptu movements, e.g., dance performance and martial arts. In addition, current research lacks fine-grained control over the generated motion, which is essential for intelligent human-computer interaction and animation creation. In this paper, we propose a novel keyframe-based mo…
University of California, Berkeley（加州大学伯克利分校）；University of Michigan（密歇根大学）；Virginia Tech（弗吉尼亚理工大学）
Neural network controllers have become popular in control tasks thanks to their flexibility and expressivity. Stability is a crucial property for safety-critical dynamical systems, while stabilization of partially observed systems, in many cases, requires controllers to retain and process long-term memories of the past. We consider the important class of recurrent neural networks (RNN) as dynamic controllers for nonlinear uncertain partially-observed systems, and derive convex stability conditions based on integral quadratic constraints, S-lemma and sequential convexification. To ensure stabil…
Technical University of Munich（慕尼黑工业大学）；Stanford University（斯坦福大学）
The modern computer graphics pipeline can synthesize images at remarkable visual quality; however, it requires well-defined, high-quality 3D content as input. In this work, we explore the use of imperfect 3D content, for instance, obtained from photo-metric reconstructions with noisy and incomplete surface geometry, while still aiming to produce photo-realistic (re-)renderings. To address this challenging problem, we introduce Deferred Neural Rendering, a new paradigm for image synthesis that combines the traditional graphics pipeline with learnable components. Specifically, we propose Neural …
Despite recent success in conditional image synthesis, prevalent input conditions such as semantics and edges are not clear enough to express `Linear (Ridges)' and `Planar (Scale)' representations. To address this problem, we propose a novel framework StyLandGAN, which synthesizes desired landscape images using a depth map which has higher expressive power. Our StyleLandGAN is extended from the unconditional generation model to accept input conditions. We also propose a '2-phase inference' pipeline which generates diverse depth maps and shifts local parts so that it can easily reflect user's i…
In this paper, we propose a method to add constraints that are un-formulatable in generative adversarial networks (GAN)-based arbitrary size RAW Bayer image generation. It is shown theoretically that by using the transformed data in GAN training, it is able to improve the learning of the original data distribution, owing to the invariant of Jensen-Shannon (JS) divergence between two distributions under invertible and differentiable transformation. Benefiting from the proposed method, RAW Bayer pattern images can be generated by configuring the transformation as demosaicing. It is shown that by…
University of Southern California（南加州大学）；Hood College（胡德学院）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
Machine learning (ML) is revolutionizing protein structural analysis, including an important subproblem of predicting protein residue contact maps, i.e., which amino-acid residues are in close spatial proximity given the amino-acid sequence of a protein. Despite recent progresses in ML-based protein contact prediction, predicting contacts with a wide range of distances (commonly classified into short-, medium- and long-range contacts) remains a challenge. Here, we propose a multiscale graph neural network (GNN) based approach taking a cue from multiscale physics simulations, in which a standar…
Data Science Institute, Imperial College London（伦敦帝国理工学院数据科学研究所）
Translating information between text and image is a fundamental problem in artificial intelligence that connects natural language processing and computer vision. In the past few years, performance in image caption generation has seen significant improvement through the adoption of recurrent neural networks (RNN). Meanwhile, text-to-image generation begun to generate plausible images using datasets of specific categories like birds and flowers. We've even seen image generation from multi-category datasets such as the Microsoft Common Objects in Context (MSCOCO) through the use of generative adv…
In this paper, we propose a novel approach to convert given speech audio to a photo-realistic speaking video of a specific person, where the output video has synchronized, realistic, and expressive rich body dynamics. We achieve this by first generating 3D skeleton movements from the audio sequence using a recurrent neural network (RNN), and then synthesizing the output video via a conditional generative adversarial network (GAN). To make the skeleton movement realistic and expressive, we embed the knowledge of an articulated 3D human skeleton and a learned dictionary of personal speech iconic…
DiffSynth: Latent In-Iteration Deflickering for Realistic Video Synthesis
School of Data Science and Engineering, East China Normal University（华东师范大学数据科学与工程学院）；School of Information, Xiamen University（厦门大学信息学院）；Alibaba Group（阿里巴巴集团）
Huazhong University of Science and Technology（华中科技大学）；Megvii Technology Inc.（旷视科技有限公司）
In this paper, we investigate the Chinese calligraphy synthesis problem: synthesizing Chinese calligraphy images with specified style from standard font(eg. Hei font) images (Fig. 1(a)). Recent works mostly follow the stroke extraction and assemble pipeline which is complex in the process and limited by the effect of stroke extraction. We treat the calligraphy synthesis problem as an image-to-image translation problem and propose a deep neural network based model which can generate calligraphy images from standard font images directly. Besides, we also construct a large scale benchmark that co…
Robust Roadside Perception: an Automated Data Synthesis Pipeline Minimizing Human Annotation
University of Michigan（密歇根大学）；University of Michigan Transportation Research Institute（密歇根大学交通研究所）；Beihang University（北京航空航天大学）；Mcity（Mcity（密歇根大学智能城市实验室））
University of British Columbia（不列颠哥伦比亚大学）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；Body Labs Inc.（Body Labs公司）
Human motion modelling is a classical problem at the intersection of graphics and computer vision, with applications spanning human-computer interaction, motion synthesis, and motion prediction for virtual and augmented reality. Following the success of deep learning methods in several computer vision tasks, recent work has focused on using deep recurrent neural networks (RNNs) to model human motion, with the goal of learning time-dependent representations that perform tasks such as short-term motion prediction and long-term human motion synthesis. We examine recent work, with a focus on the e…
Deep generative models have shown great promise when it comes to synthesising novel images. While they can generate images that look convincing on a higher-level, generating fine-grained details is still a challenge. In order to foster research on more powerful generative approaches, this paper proposes a novel task: generative modelling of 2D tree skeletons. Trees are an interesting shape class because they exhibit complexity and variations that are well-suited to measure the ability of a generative model to generated detailed structures. We propose a new dataset for this task and demonstrate…
We present a new, fast and flexible pipeline for indoor scene synthesis that is based on deep convolutional generative models. Our method operates on a top-down image-based representation, and inserts objects iteratively into the scene by predicting their category, location, orientation and size with separate neural network modules. Our pipeline naturally supports automatic completion of partial scenes, as well as synthesis of complete scenes. Our method is significantly faster than the previous image-based method and generates result that outperforms it and other state-of-the-art deep generat…
Generating realistic motions for digital humans is time-consuming for many graphics applications. Data-driven motion synthesis approaches have seen solid progress in recent years through deep generative models. These results offer high-quality motions but typically suffer in motion style diversity. For the first time, we propose a framework using the denoising diffusion probabilistic model (DDPM) to synthesize styled human motions, integrating two tasks into one pipeline with increased style diversity compared with traditional motion synthesis methods. Experimental results show that our system…
Visual Car Brand Classification by Implementing a Synthetic Image Dataset Creation Pipeline
TH OWL University of Applied Sciences and Arts（TH OWL应用科学与艺术大学）；Fraunhofer IOSB-INA（弗劳恩霍夫IOSB-INA研究所）；Institute for Industrial Information Technology (inIT)（工业信息技术研究所(inIT)）
Enhancing Prostate Cancer Diagnosis with Deep Learning: A Study using mpMRI Segmentation and Classification
KLS Gogte Institute of Technology（KLS戈格特理工学院）；University of Stavanger（斯塔万格大学）；Dr. M.S. Sheshgiri College of Engineering and Technology（M.S.谢什吉里博士工程技术学院）；JN Medical College（JN医学院）
Training data is the key ingredient for deep learning approaches, but difficult to obtain for the specialized domains often encountered in robotics. We describe a synthesis pipeline capable of producing training data for cluttered scene perception tasks such as semantic segmentation, object detection, and correspondence or pose estimation. Our approach arranges object meshes in physically realistic, dense scenes using physics simulation. The arranged scenes are rendered using high-quality rasterization with randomized appearance and material parameters. Noise and other transformations introduc…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）
Data-driven methods such as convolutional neural networks (CNNs) are known to deliver state-of-the-art performance on image recognition tasks when the training data are abundant. However, in some instances, such as change detection in remote sensing images, annotated data cannot be obtained in sufficient quantities. In this work, we propose a simple and efficient method for creating realistic targeted synthetic datasets in the remote sensing domain, leveraging the opportunities offered by game development engines. We provide a description of the pipeline for procedural geometry generation and …
INFN Section of Naples（意大利国家核物理研究所那不勒斯分部）；University of Naples "Federico II"（那不勒斯费德里科二世大学）；ESO（欧洲南方天文台）
We present a Deep-Learning (DL) pipeline developed for the detection and characterization of astronomical sources within simulated Atacama Large Millimeter/submillimeter Array (ALMA) data cubes. The pipeline is composed of six DL models: a Convolutional Autoencoder for source detection within the spatial domain of the integrated data cubes, a Recurrent Neural Network (RNN) for denoising and peak detection within the frequency domain, and four Residual Neural Networks (ResNets) for source characterization. The combination of spatial and frequency information improves completeness while decreasi…
HKUST（香港科技大学）；University of Maryland, College Park（马里兰大学帕克分校）；Zhejiang University（浙江大学）；Alibaba A.I. Labs（阿里巴巴人工智能实验室）；City University of Hong Kong（香港城市大学）
Freehand sketching is a dynamic process where points are sequentially sampled and grouped as strokes for sketch acquisition on electronic devices. To recognize a sketched object, most existing methods discard such important temporal ordering and grouping information from human and simply rasterize sketches into binary images for classification. In this paper, we propose a novel single-branch attentive network architecture RNN-Rasterization-CNN (Sketch-R2CNN for short) to fully leverage the dynamics in sketches for recognition. Sketch-R2CNN takes as input only a vector sketch with grouped seque…
