首页/论文/1908.07262

arXiv 1908.072622019-08cs.CV

A Neural Virtual Anchor Synthesizer based on Seq2Seq and GAN Models

Clouminds

Zipeng Wang, Zhaoxiang Liu, Zezhou Chen, Huan Hu, Shiguo Lian

备注：: Accepted to ISMAR 2019

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

This paper presents a novel framework to generate realistic face video of an anchor, who is reading certain news. This task is also known as Virtual Anchor. Given some paragraphs of words, we first utilize a pretrained Word2Vec model to embed each word into a vector; then we utilize a Seq2Seq-based model to translate these word embeddings into action units and head poses of the target anchor; these action units and head poses will be concatenated with facial landmarks as well as the former $n$ synthesized frames, and the concatenation serves as input of a Pix2PixHD-based model to synthesize realistic facial images for the virtual anchor. The experimental results demonstrate our framework is feasible for the synthesis of virtual anchor.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Spatial Fusion GAN for Image Synthesis

Nanyang Technological University（南洋理工大学）；Institute for Infocomm, A*STAR（新加坡科技研究局资讯通信研究院）

Recent advances in generative adversarial networks (GANs) have shown great potentials in realistic image synthesis whereas most existing works address synthesis realism in either appearance space or geometry space but few in both. This paper presents an innovative Spatial Fusion GAN (SF-GAN) that combines a geometry synthesizer and an appearance synthesizer to achieve synthesis realism in both geometry and appearance spaces. The geometry synthesizer learns contextual geometries of background images and transforms and places foreground objects into the background images unanimously. The appeara…

2019-04-02cs.CV

Commonality in Natural Images Rescues GANs: Pretraining GANs with Generic and Privacy-free Synthetic Data

Yonsei University（延世大学）

Transfer learning for GANs successfully improves generation performance under low-shot regimes. However, existing studies show that the pretrained model using a single benchmark dataset is not generalized to various target datasets. More importantly, the pretrained model can be vulnerable to copyright or privacy risks as membership inference attack advances. To resolve both issues, we propose an effective and unbiased data synthesizer, namely Primitives-PS, inspired by the generic characteristics of natural images. Specifically, we utilize 1) the generic statistics on the frequency magnitude s…

2022-04-11cs.AIcs.CV

PTNet: A High-Resolution Infant MRI Synthesizer Based on Transformer

Columbia University（哥伦比亚大学）；Columbia University Irving Medical Center（哥伦比亚大学欧文医学中心）

Magnetic resonance imaging (MRI) noninvasively provides critical information about how human brain structures develop across stages of life. Developmental scientists are particularly interested in the first few years of neurodevelopment. Despite the success of MRI collection and analysis for adults, it is a challenge for researchers to collect high-quality multimodal MRIs from developing infants mainly because of their irregular sleep pattern, limited attention, inability to follow instructions to stay still, and a lack of analysis approaches. These challenges often lead to a significant reduc…

2021-05-28cs.CVcs.LGeess.IV

A Realistic Face-to-Face Conversation System based on Deep Neural Networks

To improve the experiences of face-to-face conversation with avatar, this paper presents a novel conversation system. It is composed of two sequence-to-sequence models respectively for listening and speaking and a Generative Adversarial Network (GAN) based realistic avatar synthesizer. The models exploit the facial action and head pose to learn natural human reactions. Based on the models' output, the synthesizer uses the Pixel2Pixel model to generate realistic facial images. To show the improvement of our system, we use a 3D model based avatar driving scheme as a reference. We train and evalu…

2019-08-21cs.CVcs.SDeess.AS

Few-shot Face Image Translation via GAN Prior Distillation

Xidian University（西安电子科技大学）；University of Science and Technology of China（中国科学技术大学）

Face image translation has made notable progress in recent years. However, when training on limited data, the performance of existing approaches significantly declines. Although some studies have attempted to tackle this problem, they either failed to achieve the few-shot setting (less than 10) or can only get suboptimal results. In this paper, we propose GAN Prior Distillation (GPD) to enable effective few-shot face image translation. GPD contains two models: a teacher network with GAN Prior and a student network that fulfills end-to-end translation. Specifically, we adapt the teacher network…

2023-01-28cs.CV

Quantized GAN for Complex Music Generation from Dance Videos

Illinois Institute of Technology（伊利诺伊理工学院）；Snap Inc.（Snap公司）；Princeton University（普林斯顿大学）

We present Dance2Music-GAN (D2M-GAN), a novel adversarial multi-modal framework that generates complex musical samples conditioned on dance videos. Our proposed framework takes dance video frames and human body motions as input, and learns to generate music samples that plausibly accompany the corresponding input. Unlike most existing conditional music generation works that generate specific types of mono-instrumental sounds using symbolic audio representations (e.g., MIDI), and that usually rely on pre-defined musical synthesizers, in this work we generate dance music in complex styles (e.g.,…

2022-07-19cs.CVcs.SDeess.AS

更多相关论文收起更多论文

基于潜在空间锚定的领域可扩展非配对图像转换

Domain-Scalable Unpaired Image Translation via Latent Space Anchoring

John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）；Amazon Alexa Science（亚马逊Alexa科学部）；Computer Science Department, Boston College（波士顿学院计算机科学系）

提出一种名为潜在空间锚定的领域可扩展非配对图像转换方法，通过将不同领域图像锚定到冻结GAN的同一潜在空间，实现无需微调现有编解码器即可高效扩展至新领域并在任意领域间转换图像。

2023-06-26cs.CV

可在单GPU上合成16K图像的单样本超高分辨率生成对抗网络

One-shot Ultra-high-Resolution Generative Adversarial Network That Synthesizes 16K Images On A Single GPU

Handong Global university（韩东全球大学）

本研究提出OUR-GAN框架，仅需单张训练图像和单张消费级GPU即可生成无重复16K超高分辨率图像，通过子区域超分辨率、垂直位置卷积等技术，在保真度、一致性和多样性上优于基线模型。

2023-08-28cs.CVcs.LGeess.IV

HumanGen: Generating Human Radiance Fields with Explicit Priors

ShanghaiTech University（上海科技大学）；University of Toronto（多伦多大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）

Recent years have witnessed the tremendous progress of 3D GANs for generating view-consistent radiance fields with photo-realism. Yet, high-quality generation of human radiance fields remains challenging, partially due to the limited human-related priors adopted in existing methods. We present HumanGen, a novel 3D human generation scheme with detailed geometry and $\text{360}^{\circ}$ realistic free-view rendering. It explicitly marries the 3D human generation with various priors from the 2D generator and 3D reconstructor of humans through the design of "anchor image". We introduce a hybrid fe…

2022-12-10cs.CV

CVEGAN：一种受感知启发的用于压缩视频增强的GAN

CVEGAN: A Perceptually-inspired GAN for Compressed Video Enhancement

Bristol Vision Institute, University of Bristol（布里斯托大学布里斯托视觉研究所）

本文提出CVEGAN，通过Mul2Res、ERNB、ECBAM及ReSphereGAN感知训练增强压缩视频，并在HM16.20的PP和SRA任务中取得显著编码增益。

2020-11-26cs.CVeess.IV

Hierarchical Amortized Training for Memory-efficient High Resolution 3D GAN

University of Pittsburgh（匹兹堡大学）；The University of Melbourne（墨尔本大学）

Generative Adversarial Networks (GAN) have many potential medical imaging applications, including data augmentation, domain adaptation, and model explanation. Due to the limited memory of Graphical Processing Units (GPUs), most current 3D GAN models are trained on low-resolution medical images, these models either cannot scale to high-resolution or are prone to patchy artifacts. In this work, we propose a novel end-to-end GAN architecture that can generate high-resolution 3D images. We achieve this goal by using different configurations between training and inference. During training, we adopt…

2022-09-12cs.CVeess.IV

MAP-Diff: 多锚点引导的扩散模型用于渐进式三维全身低剂量PET去噪

MAP-Diff: Multi-Anchor Guided Diffusion for Progressive 3D Whole-Body Low-Dose PET Denoising

School of Engineering, Zurich University of Applied Sciences, CH；Bioengineering Department；Imperial-X, Imperial College London, UK；DAMTP, University of Cambridge, UK；Lucerne University Teaching；Research Hospital, CH；Lung Institute, Imperial College London, UK；Cardiovascular Research Centre, Royal Brompton Hospital, UK；School of Biomedical Engineering \& Imaging Sciences, King's College London, UK；Yau Mathematical Sciences Center, Tsinghua University, CN

MAP-Diff通过多锚点引导的扩散模型实现低剂量PET图像的渐进式去噪，提升PSNR和SSIM，降低NMAE，优于多种基线方法。

2026-03-02cs.AIcs.CV

CLERF: 面向全范围头部姿态估计的对比学习

CLERF: Contrastive LEaRning for Full Range Head Pose Estimation

Santa Clara University（圣克拉拉大学）；DOCOMO Innovations（docomo创新公司）

针对头部姿态估计数据稀疏、跨分布预测不准的问题，提出基于3D-aware GAN三元组采样的对比学习框架CLERF，实现全范围精准头部姿态估计，性能优于现有模型。

2024-12-03cs.CV

地形扩散网络：基于地质草图引导的气候感知地形生成

Terrain Diffusion Network: Climatic-Aware Terrain Generation with Geological Sketch Guidance

The University of Sydney（悉尼大学）；Civil Aviation Flight University of China（中国民用航空飞行学院）

针对现有地形生成方法难以兼顾用户控制与生成多样性的问题，本文提出地形扩散网络（TDN），采用多层级去噪方案并融合地质草图引导，在NASA地形图像数据集上实现了最先进的逼真地形生成性能。

2023-08-31cs.AIcs.CVcs.MM