We study the intrinsic transformation of feature maps across convolutional network layers with explicit top-down control. To this end, we develop top-down feature transformer (TFT), under controllable parameters, that are able to account for the hidden layer transformation while maintaining the overall consistency across layers. The learned generators capture the underlying feature transformation processes that are independent of particular training images. Our proposed TFT framework brings insights to and helps the understanding of, an important problem of studying the CNN internal feature representation and transformation under the top-down processes. In the case of spatial transformations, we demonstrate the significant advantage of TFT over existing data-driven approaches in building data-independent transformations. We also show that it can be adopted in other applications such as data augmentation and image style transfer.
We focus on controllable disentangled representation learning (C-Dis-RL), where users can control the partition of the disentangled latent space to factorize dataset attributes (concepts) for downstream tasks. Two general problems remain under-explored in current methods: (1) They lack comprehensive disentanglement constraints, especially missing the minimization of mutual information between different attributes across latent and observation domains. (2) They lack convexity constraints, which is important for meaningfully manipulating specific attributes for downstream tasks. To encourage bot…
Recent studies have shown that StyleGANs provide promising prior models for downstream tasks on image synthesis and editing. However, since the latent codes of StyleGANs are designed to control global styles, it is hard to achieve a fine-grained control over synthesized images. We present SemanticStyleGAN, where a generator is trained to model local semantic parts separately and synthesizes images in a compositional way. The structure and texture of different local parts are controlled by corresponding latent codes. Experimental results demonstrate that our model provides a strong disentanglem…
Controllable Video Generation by Learning the Underlying Dynamical System with Neural ODE
University of Edinburgh（爱丁堡大学）；University of Groningen（格罗宁根大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；University College London（伦敦大学学院）
CrackSegFlow: Controllable Flow Matching Synthesis for Generalizable Crack Segmentation with a 50K Image-Mask Benchmark
Department of Civil and Environmental Engineering, Grainger College of Engineering, University of Illinois Urbana--Champaign（土木与环境工程系，格拉inger工程学院，伊利诺伊大学厄巴纳-香槟分校）；Department of Computer Science, Grainger College of Engineering, University of Illinois Urbana--Champaign（计算机科学系，格拉inger工程学院，伊利诺伊大学厄巴纳-香槟分校）
Controllable Localized Face Anonymization Via Diffusion Inpainting
Center for Machine Vision and Signal Analysis (CMVS)（机器视觉与信号分析中心）；University of Oulu（奥卢大学）；ELLIS Institute Finland（芬兰ELLIS研究所）；The Machine Learning Group（机器学习小组）；UiT The Arctic University of Norway（挪威北冰洋大学）
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Tencent AI Lab（腾讯人工智能实验室）
The recent advances in deep learning have made it possible to generate photo-realistic images by using neural networks and even to extrapolate video frames from an input video clip. In this paper, for the sake of both furthering this exploration and our own interest in a realistic application, we study image-to-video translation and particularly focus on the videos of facial expressions. This problem challenges the deep neural networks by another temporal dimension comparing to the image-to-image translation. Moreover, its single input image fails most existing video generation methods that re…
SALIENT: Frequency-Aware Paired Diffusion for Controllable Long-Tail CT Detection
Trauma Radiology AI Laboratory (TRAIL)（创伤放射AI实验室）；Department of Radiology and Nuclear Medicine（放射科与核医学科）；University of Maryland School of Medicine（马里兰大学医学院）
Wuhan University（武汉大学）；Microsoft Research Asia（微软亚洲研究院）
Generating controllable videos conforming to user intentions is an appealing yet challenging topic in computer vision. To enable maneuverable control in line with user intentions, a novel video generation task, named Text-Image-to-Video generation (TI2V), is proposed. With both controllable appearance and motion, TI2V aims at generating videos from a static image and a text description. The key challenges of TI2V task lie both in aligning appearance and motion from different modalities, and in handling uncertainty in text descriptions. To address these challenges, we propose a Motion Anchor-ba…
Nanchang Hangkong University（南昌航空大学）；Dalian University of Technology（大连理工大学）；Cardiff University（卡迪夫大学）
Most of the existing audio-driven 3D facial animation methods suffered from the lack of detailed facial expression and head pose, resulting in unsatisfactory experience of human-robot interaction. In this paper, a novel pose-controllable 3D facial animation synthesis method is proposed by utilizing hierarchical audio-vertex attention. To synthesize real and detailed expression, a hierarchical decomposition strategy is proposed to encode the audio signal into both a global latent feature and a local vertex-wise control feature. Then the local and global audio features combined with vertex spati…
