Conditional GANs (cGAN), in their rudimentary form, suffer from critical drawbacks such as the lack of diversity in generated outputs and distortion between the latent and output manifolds. Although efforts have been made to improve results, they can suffer from unpleasant side-effects such as the topology mismatch between latent and output spaces. In contrast, we tackle this problem from a geometrical perspective and propose a novel training mechanism that increases both the diversity and the visual quality of a vanilla cGAN, by systematically encouraging a bi-lipschitz mapping between the latent and the output manifolds. We validate the efficacy of our solution on a baseline cGAN (i.e., Pix2Pix) which lacks diversity, and show that by only modifying its training mechanism (i.e., with our proposed Pix2Pix-Geo), one can achieve more diverse and realistic outputs on a broad set of image-to-image translation tasks. Codes are available at https://github.com/samgregoost/Rethinking-CGANs.
National University of Singapore（新加坡国立大学）；Tencent PCG（腾讯平台与内容事业群）
Vector-Quantized (VQ-based) generative models usually consist of two basic components, i.e., VQ tokenizers and generative transformers. Prior research focuses on improving the reconstruction fidelity of VQ tokenizers but rarely examines how the improvement in reconstruction affects the generation ability of generative transformers. In this paper, we surprisingly find that improving the reconstruction fidelity of VQ tokenizers does not necessarily improve the generation. Instead, learning to compress semantic features within VQ tokenizers significantly improves generative transformers' ability …
A very recent trend in generative modeling is building 3D-aware generators from 2D image collections. To induce the 3D bias, such models typically rely on volumetric rendering, which is expensive to employ at high resolutions. During the past months, there appeared more than 10 works that address this scaling issue by training a separate 2D decoder to upsample a low-resolution image (or a feature tensor) produced from a pure 3D generator. But this solution comes at a cost: not only does it break multi-view consistency (i.e. shape and texture change when the camera moves), but it also learns th…
Istanbul Technical University (ITU)（伊斯坦布尔理工大学）；ITU-AI: Artificial Intelligence and Data Science Research and Application Center（伊斯坦布尔理工大学人工智能与数据科学研究与应用中心）；ITU - CSCRS (Application and Research Center for Satellite Communications and Remote Sensing)（伊斯坦布尔理工大学卫星通信与遥感应用研究中心）
Convolutional Neural Networks (CNN)-based approaches have shown promising results in pansharpening of satellite images in recent years. However, they still exhibit limitations in producing high-quality pansharpening outputs. To that end, we propose a new self-supervised learning framework, where we treat pansharpening as a colorization problem, which brings an entirely novel perspective and solution to the problem compared to existing methods that base their solution solely on producing a super-resolution version of the multispectral image. Whereas CNN-based methods provide a reduced resolutio…
Stanford University（斯坦福大学）；Google Research（谷歌研究院）；University of California, San Diego（加州大学圣地亚哥分校）；Shanghai Jiao Tong Univeristy（上海交通大学）
In this paper, we examine the long-neglected yet important effects of point sampling patterns in point cloud GANs. Through extensive experiments, we show that sampling-insensitive discriminators (e.g.PointNet-Max) produce shape point clouds with point clustering artifacts while sampling-oversensitive discriminators (e.g.PointNet++, DGCNN) fail to guide valid shape generation. We propose the concept of sampling spectrum to depict the different sampling sensitivities of discriminators. We further study how different evaluation metrics weigh the sampling pattern against the geometry and propose s…
For successful semantic editing of real images, it is critical for a GAN inversion method to find an in-domain latent code that aligns with the domain of a pre-trained GAN model. Unfortunately, such in-domain latent codes can be found only for in-range images that align with the training images of a GAN model. In this paper, we propose BDInvert, a novel GAN inversion approach to semantic editing of out-of-range images that are geometrically unaligned with the training images of a GAN model. To find a latent code that is semantically editable, BDInvert inverts an input out-of-range image into a…
When Semantics Regulate: Rethinking Patch Shuffle and Internal Bias for Generated Image Detection with CLIP
School of CyberSpace Security, Beijing University of Posts and Telecommunications（信息与电子学院，北京邮电大学）；School of Finance, Central University of Finance and Economics（金融学院，中央财经大学）
University of Chinese Academy of Sciences（中国科学院大学）；JD AI Research（京东人工智能研究院）
Single image generation (SIG), described as generating diverse samples that have similar visual content with the given single image, is first introduced by SinGAN which builds a pyramid of GANs to progressively learn the internal patch distribution of the single image. It also shows great potentials in a wide range of image manipulation tasks. However, the paradigm of SinGAN has limitations in terms of generation quality and training time. Firstly, due to the lack of high-level information, SinGAN cannot handle the object images well as it does on the scene and texture images. Secondly, the se…
NEC Laboratories America（美国NEC实验室）；University of Southern California（南加州大学）；University of California, San Diego（加州大学圣地亚哥分校）
We address the challenging problem of generating facial attributes using a single image in an unconstrained pose. In contrast to prior works that largely consider generation on 2D near-frontal images, we propose a GAN-based framework to generate attributes directly on a dense 3D representation given by UV texture and position maps, resulting in photorealistic, geometrically-consistent and identity-preserving outputs. Starting from a self-occluded UV texture map obtained by applying an off-the-shelf 3D reconstruction method, we propose two novel components. First, a texture completion generativ…
Max-Planck-Institute for Informatics（马克斯-普朗克信息研究所）；Stanford University（斯坦福大学）；Universidad Rey Juan Carlos（胡安卡洛斯国王大学）
We address the highly challenging problem of real-time 3D hand tracking based on a monocular RGB-only sequence. Our tracking method combines a convolutional neural network with a kinematic 3D hand model, such that it generalizes well to unseen data, is robust to occlusions and varying camera viewpoints, and leads to anatomically plausible as well as temporally smooth hand motions. For training our CNN we propose a novel approach for the synthetic generation of training data that is based on a geometrically consistent image-to-image translation network. To be more specific, we use a neural netw…
Savoie Mont Blanc University（萨瓦勃朗峰大学）；InModelia（InModelia公司）
Geometrical and appearance quality requirements set the limits of the current industrial performance in injection molding. To guarantee the product's quality, it is necessary to adjust the process settings in a closed loop. Those adjustments cannot rely on the final quality because a part takes days to be geometrically stable. Thus, the final part geometry must be predicted from measurements on hot parts. In this paper, we use recent success of Generative Adversarial Networks (GAN) with the pix2pix network architecture to predict the final part geometry, using only hot parts thermographic imag…
RAVEN: Rethinking Adversarial Video Generation with Efficient Tri-plane Networks
Max Planck Institute for intelligent Systems（马克斯·普朗克智能系统研究所）；Inria（法国国家信息与自动化研究所）；École normale supérieure（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）；ELLIS Institute Tübingen（埃利斯研究所蒂宾根分所）
Inverse Design of Realizable Metasurface based Absorbers using Improved Conditioning and Diversity Enhanced Progressively Growing GANs
Centre for Electromagnetics, CSIR-National Aerospace Laboratories（电磁研究中心，国家航空航天实验室）；Birla Institute of Technology and Science, Pilani, Rajasthan（比拉理工学院和科学学院，比里尼）；Indian Institute of Technology, Bombay, Maharashtra（孟买印度理工学院，马哈拉施特拉）
Monocular depth estimation is an extensively studied computer vision problem with a vast variety of applications. Deep learning-based methods have demonstrated promise for both supervised and unsupervised depth estimation from monocular images. Most existing approaches treat depth estimation as a regression problem with a local pixel-wise loss function. In this work, we innovate beyond existing approaches by using adversarial training to learn a context-aware, non-local loss function. Such an approach penalizes the joint configuration of predicted depth values at the patch-level instead of the…
Rethinking Foundation Model Collaboration: Enhancing Specialized Models through Proxy Task Reasoning
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与运载学院）；Department of Urban Studies and Planning, Massachusetts Institute of Technology（麻省理工学院城市研究与规划系）
A biased dataset is a dataset that generally has attributes with an uneven class distribution. These biases have the tendency to propagate to the models that train on them, often leading to a poor performance in the minority class. In this project, we will explore the extent to which various data augmentation methods alleviate intrinsic biases within the dataset. We will apply several augmentation techniques on a sample of the UTKFace dataset, such as undersampling, geometric transformations, variational autoencoders (VAEs), and generative adversarial networks (GANs). We then trained a classif…
Deformation-Invariant Neural Network and Its Applications in Distorted Image Restoration and Analysis
City University of Hong Kong（香港城市大学）；Hong Kong Center for Cerebro-Cardiovascular Health Engineering（香港脑心血管健康工程中心）；Chinese University of Hong Kong（香港中文大学）
