We present a 3D modeling method which enables end-users to refine or detailize 3D shapes using machine learning, expanding the capabilities of AI-assisted 3D content creation. Given a coarse voxel shape (e.g., one produced with a simple box extrusion tool or via generative modeling), a user can directly "paint" desired target styles representing compelling geometric details, from input exemplar shapes, over different regions of the coarse shape. These regions are then up-sampled into high-resolution geometries which adhere with the painted styles. To achieve such controllable and localized 3D detailization, we build on top of a Pyramid GAN by making it masking-aware. We devise novel structural losses and priors to ensure that our method preserves both desired coarse structures and fine-grained features even if the painted styles are borrowed from diverse sources, e.g., different semantic parts and even different shape categories. Through extensive experiments, we show that our ability to localize details enables novel interactive creative workflows and applications. Our experiments further demonstrate that in comparison to prior techniques built on global detailization, our method generates structure-preserving, high-resolution stylized geometries with more coherent shape details and style transitions.
Controllable Localized Face Anonymization Via Diffusion Inpainting
Center for Machine Vision and Signal Analysis (CMVS)（机器视觉与信号分析中心）；University of Oulu（奥卢大学）；ELLIS Institute Finland（芬兰ELLIS研究所）；The Machine Learning Group（机器学习小组）；UiT The Arctic University of Norway（挪威北冰洋大学）
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
The success of Deep Generative Models at high-resolution image generation has led to their extensive utilization for style editing of real images. Most existing methods work on the principle of inverting real images onto their latent space, followed by determining controllable directions. Both inversion of real images and determination of controllable latent directions are computationally expensive operations. Moreover, the determination of controllable latent directions requires additional human supervision. This work aims to explore the efficacy of mask-guided feature modulation in the laten…
Recently, methods have been proposed that perform texture synthesis and style transfer by using convolutional neural networks (e.g. Gatys et al. [2015,2016]). These methods are exciting because they can in some cases create results with state-of-the-art quality. However, in this paper, we show these methods also have limitations in texture quality, stability, requisite parameter tuning, and lack of user controls. This paper presents a multiscale synthesis pipeline based on convolutional neural networks that ameliorates these issues. We first give a mathematical explanation of the source of ins…
University of Edinburgh（爱丁堡大学）；Canon Medical Research Europe（佳能医疗欧洲研究中心）；The Alan Turing Institute（艾伦·图灵研究所）
Acquiring annotated data at scale with rare diseases or conditions remains a challenge. It would be extremely useful to have a method that controllably synthesizes images that can correct such underrepresentation. Assuming a proper latent representation, the idea of a "latent vector arithmetic" could offer the means of achieving such synthesis. A proper representation must encode the fidelity of the input data, preserve invariance and equivariance, and permit arithmetic operations. Motivated by the ability to disentangle images into spatial anatomy (tensor) factors and accompanying imaging (ve…
Trajectory Forcing: Structure-First Generation with Controllable Semantic Trajectories
University of Tübingen（图宾根大学）；ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ELLIS Institute（ELLIS研究所）；Tübingen AI Center（图宾根人工智能中心）
MoGen: A Unified Collaborative Framework for Controllable Multi-Object Image Generation
Faculty of Applied Sciences, Macao Polytechnic University（澳门理工学院）；College of Computer Science, Sichuan University（四川大学计算机学院）；Department of Computer Science and Engineering, Michigan State University（密歇根州立大学计算机科学与工程系）；School of Information Science and Electronic Engineering, Shanghai Jiao Tong University（上海交通大学信息科学与电子工程学院）；School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）
QCAgent: An agentic framework for quality-controllable pathology report generation from whole slide image
School of Biomedical Engineering, Division of Life Sciences；Medicine, University of Science；Technology of China, Hefei, Anhui, 230026, P.R. China Center for Medical Imaging, Robotics, Analytic Computing \& Learning (MIRACLE), Suzhou Institute for Advance Research, USTC, 215123, P.R. China Chinese PLA General Hospital Ninth Medical Center, China Hong Kong University of Science
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Huazhong University of Science and Technology（华中科技大学）；Yunnan Open University（云南开放大学）；University of Electronic Science and Technology of China（电子科技大学）
In this paper, we propose a novel Deep Localized Makeup Transfer Network to automatically recommend the most suitable makeup for a female and synthesis the makeup on her face. Given a before-makeup face, her most suitable makeup is determined automatically. Then, both the beforemakeup and the reference faces are fed into the proposed Deep Transfer Network to generate the after-makeup face. Our end-to-end makeup transfer network have several nice properties including: (1) with complete functions: including foundation, lip gloss, and eye shadow transfer; (2) cosmetic specific: different cosmetic…
