Modern learning-based approaches to 3D-aware image synthesis achieve high photorealism and 3D-consistent viewpoint changes for the generated images. Existing approaches represent instances in a shared canonical space. However, for in-the-wild datasets a shared canonical system can be difficult to define or might not even exist. In this work, we instead model instances in view space, alleviating the need for posed images and learned camera distributions. We find that in this setting, existing GAN-based methods are prone to generating flat geometry and struggle with distribution coverage. We hence propose WildFusion, a new approach to 3D-aware image synthesis based on latent diffusion models (LDMs). We first train an autoencoder that infers a compressed latent representation, which additionally captures the images' underlying 3D structure and enables not only reconstruction but also novel view synthesis. To learn a faithful 3D representation, we leverage cues from monocular depth prediction. Then, we train a diffusion model in the 3D-aware latent space, thereby enabling synthesis of high-quality 3D-consistent image samples, outperforming recent state-of-the-art GAN-based methods. Importantly, our 3D-aware LDM is trained without any direct supervision from multiview images or 3D geometry and does not require posed images or learned pose or camera distributions. It directly learns a 3D representation without relying on canonical camera coordinates. This opens up promising research avenues for scalable 3D-aware image synthesis and 3D content creation from in-the-wild image data. See https://katjaschwarz.github.io/wildfusion for videos of our 3D results.
Although recent gaze estimation methods lay great emphasis on attentively extracting gaze-relevant features from facial or eye images, how to define features that include gaze-relevant components has been ambiguous. This obscurity makes the model learn not only gaze-relevant features but also irrelevant ones. In particular, it is fatal for the cross-dataset performance. To overcome this challenging issue, we propose a gaze-aware analytic manipulation method, based on a data-driven approach with generative adversarial network inversion's disentanglement characteristics, to selectively utilize g…
University of Science and Technology of China（中国科学技术大学）
Generative adversarial networks have been widely used in image synthesis in recent years and the quality of the generated image has been greatly improved. However, the flexibility to control and decouple facial attributes (e.g., eyes, nose, mouth) is still limited. In this paper, we propose a novel approach, called ChildGAN, to generate a child's image according to the images of parents with heredity prior. The main idea is to disentangle the latent space of a pre-trained generation model and precisely control the face attributes of child images with clear semantics. We use distances between f…
Robust Dreamer: Deviation-Aware Latent Gaussian Memory for Action-Controlled AR Video Generation
School of Computing, National University of Singapore（新加坡国立大学计算机学院）；Technische Universität München（慕尼黑技术大学）；Vertex Lab（Vertex实验室）；Australian National University（澳大利亚国立大学）
Recent works have shown that a rich set of semantic directions exist in the latent space of Generative Adversarial Networks (GANs), which enables various facial attribute editing applications. However, existing methods may suffer poor attribute variation disentanglement, leading to unwanted change of other attributes when altering the desired one. The semantic directions used by existing methods are at attribute level, which are difficult to model complex attribute correlations, especially in the presence of attribute distribution bias in GAN's training set. In this paper, we propose a novel f…
Topology-Aware Latent Diffusion for 3D Shape Generation
Nanyang Technological University（南洋理工大学）；Dalian University of Technology（大连理工大学）；Fudan University（复旦大学）；Chinese Academy of Sciences（中国科学院）；SenseTime Research（商汤科技研究院）
KDC-Diff: A Latent-Aware Diffusion Model with Knowledge Retention for Memory-Efficient Image Generation
The University of Queensland（昆士兰大学）；Rajshahi University of Engineering & Technology（拉贾沙希工程与技术大学）；Imam Mohammad Ibn Saud Islamic University (IMSIU)（伊玛姆穆罕默德·本·沙特伊斯兰大学）；Charles Sturt University（查尔斯·斯特劳特大学）
Department of Electrical Engineering, Indian Institute of Technology Madras (IITM)（电子工程系，印度理工学院Madras（IITM））；Sudha Gopalakrishnan Brain Centre (SGBC), IITM（Sudha Gopalakrishnan脑中心（SGBC），IITM）
FDS: Frequency-Aware Denoising Score for Text-Guided Latent Diffusion Image Editing
School of Computer and Communication Sciences, EPFL（EPFL计算机与通信科学学院）；Department of Electrical and Photonics Engineering, DTU（丹麦技术大学电气与光子工程系）；Department of Electrical Engineering, Chalmers University of Technology（查尔姆斯理工大学电气工程系）
In this paper, we explore Latent Dirichlet Allocation (LDA) and Polylingual Latent Dirichlet Allocation (PolyLDA), as a means to discover trending styles in Overstock from deep visual semantic features transferred from a pretrained convolutional neural network and text-based item attributes. To utilize deep visual semantic features in conjunction with LDA, we develop a method for creating a bag of words representation of unrolled image vectors. By viewing the channels within the convolutional layers of a Resnet-50 as being representative of a word, we can index these activations to create visu…
Textual-visual matching aims at measuring similarities between sentence descriptions and images. Most existing methods tackle this problem without effectively utilizing identity-level annotations. In this paper, we propose an identity-aware two-stage framework for the textual-visual matching problem. Our stage-1 CNN-LSTM network learns to embed cross-modal features with a novel Cross-Modal Cross-Entropy (CMCE) loss. The stage-1 network is able to efficiently screen easy incorrect matchings and also provide initial training point for the stage-2 training. The stage-2 CNN-LSTM network refines th…
DriveFuture: Future-Aware Latent World Models for Autonomous Driving
Beijing Institute of Technology（北京理工大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beihang University（北航）；Beijing Jiaotong University（北京交通大学）；The University of Queensland（昆士兰大学）；University of Macau（澳门大学）；Nanyang Technological University（南洋理工大学）；School of Artificial Intelligence ( School of Software), Yanshan University（燕山大学人工智能学院（软件学院））
Universidade Federal de Minas Gerais (UFMG)（米纳斯吉拉斯联邦大学 (UFMG)）
Many of the state-of-the-art algorithms for gesture recognition are based on Conditional Random Fields (CRFs). Successful approaches, such as the Latent-Dynamic CRFs, extend the CRF by incorporating latent variables, whose values are mapped to the values of the labels. In this paper we propose a novel methodology to set the latent values according to the gesture complexity. We use an heuristic that iterates through the samples associated with each label value, stimating their complexity. We then use it to assign the latent values to the label values. We evaluate our method on the task of recog…
Korea Advanced Institute of Science and Technology（韩国科学技术院）
Editing hairstyle is unique and challenging due to the complexity and delicacy of hairstyle. Although recent approaches significantly improved the hair details, these models often produce undesirable outputs when a pose of a source image is considerably different from that of a target hair image, limiting their real-world applications. HairFIT, a pose-invariant hairstyle transfer model, alleviates this limitation yet still shows unsatisfactory quality in preserving delicate hair textures. To solve these limitations, we propose a high-performing pose-invariant hairstyle transfer model equipped …
Echo-DM: Ultrasound Marker Removal via Conditional Latent Diffusion and Region-Aware Fusion
School of Computer Science, Wuhan University, China（武汉大学计算机学院）；The Central Hospital of Wuhan, China（武汉市中心医院）；School of Computer Science, Hubei University of Technology, China（湖北工业大学计算机学院）
We integrate two powerful ideas, geometry and deep visual representation learning, into recurrent network architectures for mobile visual scene understanding. The proposed networks learn to "lift" and integrate 2D visual features over time into latent 3D feature maps of the scene. They are equipped with differentiable geometric operations, such as projection, unprojection, egomotion estimation and stabilization, in order to compute a geometrically-consistent mapping between the world scene and their 3D latent feature state. We train the proposed architectures to predict novel camera views give…
Adaptive Augmentation-Aware Latent Learning for Robust LiDAR Semantic Segmentation
University of Science and Technology of China（中国科学技术大学）；National Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory（深空探测全国重点实验室（深空探测实验室））
University of Moratuwa（莫拉图瓦大学）；The University of Sydney（悉尼大学）
3D generative models have been recently successful in generating realistic 3D objects in the form of point clouds. However, most models do not offer controllability to manipulate the shape semantics of component object parts without extensive semantic attribute labels or other reference point clouds. Moreover, beyond the ability to perform simple latent vector arithmetic or interpolations, there is a lack of understanding of how part-level semantics of 3D shapes are encoded in their corresponding generative latent spaces. In this paper, we propose 3DLatNav; a novel approach to navigating pretr…
We present recurrent geometry-aware neural networks that integrate visual information across multiple views of a scene into 3D latent feature tensors, while maintaining an one-to-one mapping between 3D physical locations in the world scene and latent feature locations. Object detection, object segmentation, and 3D reconstruction is then carried out directly using the constructed 3D feature memory, as opposed to any of the input 2D images. The proposed models are equipped with differentiable egomotion-aware feature warping and (learned) depth-aware unprojection operations to achieve geometrical…
FLARE: Learning Future-Aware Latent Representations from Vision-Language Models for Autonomous Driving
Shanghai Jiaotong University（上海交通大学）；Shanghai Innovation Institute（上海创新研究院）；OpenDriveLab at The University of Hong Kong（香港大学OpenDrive实验室）；Li Auto Inc.（Li汽车公司）
Unmanned aerial vehicle (UAV) tracking is crucial for autonomous navigation and has broad applications in robotic automation fields. However, reliable UAV tracking remains a challenging task due to various difficulties like frequent occlusion and aspect ratio change. Additionally, most of the existing work mainly focuses on explicit information to improve tracking performance, ignoring potential interframe connections. To address the above issues, this work proposes a novel framework with continuity-aware latent interframe information mining for reliable UAV tracking, i.e., ClimRT. Specificall…
Disentangled Human Body Representation Based on Unsupervised Semantic-Aware Learning
X-ray Product, Siemens Shanghai Medical Equipment Ltd.（西门子上海医疗设备有限公司X射线产品部）；University of Science and Technology of China, School of Biomedical Engineering & Suzhou Institute for Advanced Research（中国科学技术大学生物医学工程学院及苏州先进研究院）
LRA-GNN: Latent Relation-Aware Graph Neural Network with Initial and Dynamic Residual for Facial Age Estimation
College of Computer and Mathematics, Central South University of Forestry and Technology（中南林业科技大学计算机与数学学院）；Department of Computer Science, State University of New York（纽约州立大学计算机科学系）
