Devising indicative evaluation metrics for the image generation task remains an open problem. The most widely used metric for measuring the similarity between real and generated images has been the Fréchet Inception Distance (FID) score. Because it does not differentiate the fidelity and diversity aspects of the generated images, recent papers have introduced variants of precision and recall metrics to diagnose those properties separately. In this paper, we show that even the latest version of the precision and recall metrics are not reliable yet. For example, they fail to detect the match between two identical distributions, they are not robust against outliers, and the evaluation hyperparameters are selected arbitrarily. We propose density and coverage metrics that solve the above issues. We analytically and experimentally show that density and coverage provide more interpretable and reliable signals for practitioners than the existing metrics. Code: https://github.com/clovaai/generative-evaluation-prdc.
Generative models are increasingly able to produce remarkably high quality images and text. The community has developed numerous evaluation metrics for comparing generative models. However, these metrics do not effectively quantify data diversity. We develop a new diversity metric that can readily be applied to data, both synthetic and natural, of any type. Our method employs random network distillation, a technique introduced in reinforcement learning. We validate and deploy this metric on both images and text. We further explore diversity in few-shot image generation, a setting which was pre…
In recent years, the use of Generative Adversarial Networks (GANs) has become very popular in generative image modeling. While style-based GAN architectures yield state-of-the-art results in high-fidelity image synthesis, computationally, they are highly complex. In our work, we focus on the performance optimization of style-based generative models. We analyze the most computationally hard parts of StyleGAN2, and propose changes in the generator network to make it possible to deploy style-based generative networks in the edge devices. We introduce MobileStyleGAN architecture, which has x3.5 fe…
Conditional Vendi Score: Prompt-Aware Diversity Evaluation for Generative AI Models and LLMs
Department of Computer Science and Engineering, The Chinese University of Hong Kong（计算机科学与工程系，香港中文大学）；Department of Information Engineering, The Chinese University of Hong Kong（信息工程系，香港中文大学）
针对文本提示引导的生成模型，提出条件 Vendi 和条件 RKE 分数，通过条件熵分离模型自身多样性，并证明收敛性及在多个任务中恢复真实多样性排序。
On the Fairness, Diversity and Reliability of Text-to-Image Generative Models
University of Western Australia（西澳大学）；University of Melbourne（墨尔本大学）；University of British Columbia（不列颠哥伦比亚大学）；Australian National University（澳大利亚国立大学）
Rice University（莱斯大学）；Meta AI Research（Meta人工智能研究院）
We present Polarity Sampling, a theoretically justified plug-and-play method for controlling the generation quality and diversity of pre-trained deep generative networks DGNs). Leveraging the fact that DGNs are, or can be approximated by, continuous piecewise affine splines, we derive the analytical DGN output space distribution as a function of the product of the DGN's Jacobian singular values raised to a power $ρ$. We dub $ρ$ the $\textbf{polarity}$ parameter and prove that $ρ$ focuses the DGN sampling on the modes ($ρ< 0$) or anti-modes ($ρ> 0$) of the DGN output-space distribution. We demo…
Seoul National University（首尔大学）；Purdue University（普渡大学）；NAVER WEBTOON
Reconstructing 3D human faces in the wild with the 3D Morphable Model (3DMM) has become popular in recent years. While most prior work focuses on estimating more robust and accurate geometry, relatively little attention has been paid to improving the quality of the texture model. Meanwhile, with the advent of Generative Adversarial Networks (GANs), there has been great progress in reconstructing realistic 2D images. Recent work demonstrates that GANs trained with abundant high-quality UV maps can produce high-fidelity textures superior to those produced by existing methods. However, acquiring …
DIMCIM: A Quantitative Evaluation Framework for Default-mode Diversity and Generalization in Text-to-Image Generative Models
FAIR at Meta - New York and Montreal labs（Meta 新 York 和 Montreal 实验室）；Courant Institute for Mathematical Sciences, NYU（纽约大学应用数学科学研究所）；Mila - Quebec AI Institute（魁北克人工智能研究所）；McGill University（麦吉尔大学）；Canada CIFAR AI chair（加拿大 CIFAR 人工智能主席）
We present a novel discriminator for GANs that improves realness and diversity of generated samples by learning a structured hypersphere embedding space using spherical circles. The proposed discriminator learns to populate realistic samples around the longest spherical circle, i.e., a great circle, while pushing unrealistic samples toward the poles perpendicular to the great circle. Since longer circles occupy larger area on the hypersphere, they encourage more diversity in representation learning, and vice versa. Discriminating samples based on their corresponding spherical circles can thus …
University of Science and Technology of China（中国科学技术大学）
In conditional Generative Adversarial Networks (cGANs), when two different initial noises are concatenated with the same conditional information, the distance between their outputs is relatively smaller, which makes minor modes likely to collapse into large modes. To prevent this happen, we proposed a hierarchical mode exploring method to alleviate mode collapse in cGANs by introducing a diversity measurement into the objective function as the regularization term. We also introduced the Expected Ratios of Expansion (ERE) into the regularization term, by minimizing the sum of differences betwee…
UltraDfeGAN: Detail-Enhancing Generative Adversarial Networks for High-Fidelity Functional Ultrasound Synthesis
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Department of Information and Communication, Institute of Technology of Cambodia（柬埔寨技术学院信息与通信系）；Department of Telecommunication and Network Engineering, Institute of Technology of Cambodia（柬埔寨技术学院电信与网络工程系）
KAIST（韩国科学技术院）；University of Michigan（密歇根大学）；Google Brain（谷歌大脑）
Learning disentangled representation of data without supervision is an important step towards improving the interpretability of generative models. Despite recent advances in disentangled representation learning, existing approaches often suffer from the trade-off between representation learning and generation performance i.e. improving generation quality sacrifices disentanglement performance). We propose an Information-Distillation Generative Adversarial Network (ID-GAN), a simple yet generic framework that easily incorporates the existing state-of-the-art models for both disentanglement lear…
A class of recent approaches for generating images, called Generative Adversarial Networks (GAN), have been used to generate impressively realistic images of objects, bedrooms, handwritten digits and a variety of other image modalities. However, typical GAN-based approaches require large amounts of training data to capture the diversity across the image modality. In this paper, we propose DeLiGAN -- a novel GAN-based architecture for diverse and limited training data scenarios. In our approach, we reparameterize the latent generative space as a mixture model and learn the mixture model's param…
VUNO Incorporation（VUNO公司）；University of Ulsan, College of Medicine（蔚山大学医学院）；Asan Medical Center（峨山医学中心）
Evaluating the performance of generative models in image synthesis is a challenging task. Although the Fréchet Inception Distance is a widely accepted evaluation metric, it integrates different aspects (e.g., fidelity and diversity) of synthesized images into a single score and assumes the normality of embedded vectors. Recent methods such as precision-and-recall and its variants such as density-and-coverage have been developed to separate fidelity and diversity based on k-nearest neighborhood methods. In this study, we propose an algorithm named barcode, which is inspired by the topological d…
Despite the success of generative adversarial networks (GANs) for image generation, the trade-off between visual quality and image diversity remains a significant issue. This paper achieves both aims simultaneously by improving the stability of training GANs. The key idea of the proposed approach is to implicitly regularize the discriminator using representative features. Focusing on the fact that standard GAN minimizes reverse Kullback-Leibler (KL) divergence, we transfer the representative feature, which is extracted from the data distribution using a pre-trained autoencoder (AE), to the dis…
Carnegie Mellon University（卡内基梅隆大学）；UC San Diego（加州大学圣地亚哥分校）
We present a new generative autoencoder model with dual contradistinctive losses to improve generative autoencoder that performs simultaneous inference (reconstruction) and synthesis (sampling). Our model, named dual contradistinctive generative autoencoder (DC-VAE), integrates an instance-level discriminative loss (maintaining the instance-level fidelity for the reconstruction/synthesis) with a set-level adversarial loss (encouraging the set-level fidelity for there construction/synthesis), both being contradistinctive. Extensive experimental results by DC-VAE across different resolutions inc…
Inverse Design of Realizable Metasurface based Absorbers using Improved Conditioning and Diversity Enhanced Progressively Growing GANs
Centre for Electromagnetics, CSIR-National Aerospace Laboratories（电磁研究中心，国家航空航天实验室）；Birla Institute of Technology and Science, Pilani, Rajasthan（比拉理工学院和科学学院，比里尼）；Indian Institute of Technology, Bombay, Maharashtra（孟买印度理工学院，马哈拉施特拉）
We extensively study how to combine Generative Adversarial Networks and learned compression to obtain a state-of-the-art generative lossy compression system. In particular, we investigate normalization layers, generator and discriminator architectures, training strategies, as well as perceptual losses. In contrast to previous work, i) we obtain visually pleasing reconstructions that are perceptually similar to the input, ii) we operate in a broad range of bitrates, and iii) our approach can be applied to high-resolution images. We bridge the gap between rate-distortion-perception theory and pr…
INRIA RITS（法国国家信息与自动化研究所RITS）；LANDING AI（澜舟科技）；Signal Ocean（信号海洋）；Philips Research（飞利浦研究院）；Eindhoven University of Technology（埃因霍温理工大学）；U2IS ENSTA Institut Polytechnique Paris（巴黎综合理工学院ENSTA U2IS实验室）；INRIA Flowers（法国国家信息与自动化研究所Flowers实验室）
Image captioning models have been able to generate grammatically correct and human understandable sentences. However most of the captions convey limited information as the model used is trained on datasets that do not caption all possible objects existing in everyday life. Due to this lack of prior information most of the captions are biased to only a few objects present in the scene, hence limiting their usage in daily life. In this paper, we attempt to show the biased nature of the currently existing image captioning models and present a new image captioning dataset, Egoshots, consisting of …
Seoul National University（首尔国立大学）；Sookmyung Women’s University（淑明女子大学）
Generative design refers to computational design methods that can automatically conduct design exploration under constraints defined by designers. Among many approaches, topology optimization-based generative designs aim to explore diverse topology designs, which cannot be represented by conventional parametric design approaches. Recently, data-driven topology optimization research has started to exploit artificial intelligence, such as deep learning or machine learning, to improve the capability of design exploration. This study proposes a reinforcement learning (RL) based generative design p…
Tel Aviv University（特拉维夫大学）；Hebrew University of Jerusalem（耶路撒冷希伯来大学）
Truncation is widely used in generative models for improving the quality of the generated samples, at the expense of reducing their diversity. We propose to leverage the StyleGAN generative architecture to devise a new truncation technique, based on a decomposition of the latent space into clusters, enabling customized truncation to be performed at multiple semantic levels. We do so by learning to re-generate W-space, the extended intermediate latent space of StyleGAN, using a learnable mixture of Gaussians, while simultaneously training a classifier to identify, for each latent vector, the cl…
High-fidelity facial avatar reconstruction from a monocular video is a significant research problem in computer graphics and computer vision. Recently, Neural Radiance Field (NeRF) has shown impressive novel view rendering results and has been considered for facial avatar reconstruction. However, the complex facial dynamics and missing 3D information in monocular videos raise significant challenges for faithful facial reconstruction. In this work, we propose a new method for NeRF-based facial avatar reconstruction that utilizes 3D-aware generative prior. Different from existing works that depe…
FlowObject: Flow Steering for Bridging Generative Priors and Reconstruction Fidelity
Graz University of Technology Austria（奥地利格拉茨理工大学）；Tampere University Finland（芬兰塔尔库大学）；Technical University of Munich Germany（德国慕尼黑技术大学）；Stanford University The United States of America（美国斯坦福大学）；Xi’an Jiaotong University China（中国西安交通大学）；École des Ponts ParisTech France（法国巴黎综合理工学院）
