UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild
Salesforce AI Research(赛富时人工智能研究院);Northeastern University(东北大学)
Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, Stefano Ermon, Yun Fu, Ran Xu
Achieving machine autonomy and human control often represent divergent objectives in the design of interactive AI systems. Visual generative foundation models such as Stable Diffusion show promise in navigating these goals, especially when prompted with arbitrary languages. However, they often fall short in generating images with spatial, structural, or geometric controls. The integration of such controls, which can accommodate various visual conditions in a single unified model, remains an unaddressed challenge. In response, we introduce UniControl, a new generative foundation model that consolidates a wide array of controllable condition-to-image (C2I) tasks within a singular framework, while still allowing for arbitrary language prompts. UniControl enables pixel-level-precise image generation, where visual conditions primarily influence the generated structures and language prompts guide the style and context. To equip UniControl with the capacity to handle diverse visual conditions, we augment pretrained text-to-image diffusion models and introduce a task-aware HyperNet to modulate the diffusion models, enabling the adaptation to different C2I tasks simultaneously. Trained on nine unique C2I tasks, UniControl demonstrates impressive zero-shot generation abilities with unseen visual conditions. Experimental results show that UniControl often surpasses the performance of single-task-controlled methods of comparable model sizes. This control versatility positions UniControl as a significant advancement in the realm of controllable visual generation.
Exploring Disentangled and Controllable Human Image Synthesis: From End-to-End to Stage-by-Stage
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen(科学与工程学院,香港中文大学(深圳));Future Network of Intelligence Institute, CUHK-Shenzhen(智能网络研究院,CUHK-深圳)
RealisDance-DiT: Simple yet Strong Baseline towards Controllable Character Animation in the Wild
DAMO Academy, Alibaba Group(达摩院,阿里巴巴集团);Zhejiang University(浙江大学);Hupan Lab(虎斑实验室);Southern University of Science and Technology(南方科技大学);Shenzhen University(深圳大学)
Car-Studio: Learning Car Radiance Fields from Single-View and Endless In-the-wild Images
The Hong Kong University of Science and Technology(香港科技大学);Tsinghua University(清华大学);The Hong Kong University of Science and Technology(Guangzhou)(香港科技大学(广州))
PSGAN++: Robust Detail-Preserving Makeup Transfer and Removal
Beihang University(北京航空航天大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);National University of Singapore(新加坡国立大学);Sea AI Lab(Sea人工智能实验室)
在本文中,我们同时解决化妆转移和去除任务,其目的是将化妆从参考图像转移到源图像和去除化妆从化妆图像分别。现有的方法在受限场景中已经取得了很大的进步,但是对于它们来说,在具有大的姿势和表情差异的图像之间转移化妆,或者处理化妆细节,如脸颊上的腮红或鼻子上的高光,仍然是非常具有挑战性的。此外,它们几乎不能控制转印期间的化妆程度或转印输入面中的指定部分。在这项工作中,我们提出了PSGAN++,它能够执行细节保留化妆转移和有效的卸妆。对于化妆转移,PSGAN++使用化妆提取网络来提取化妆信息,这些信息被嵌入到空间感知化妆矩阵中。我们还设计了一个Attentive Makeup Morphing模块,该模块指定源图像中的化妆如何从参考图像中变形,以及化妆细节损失以监督所选化妆细节区域内的模型。另一方面,对于卸妆,PSGAN++应用身份提取网络将来自化妆图像的身份信息嵌入到身份矩阵中。最后,将获得的化妆/单位矩阵馈送到风格转移网络,该网络能够编辑特征图以实现化妆转移或去除。为了评估我们的PSGAN++的有效性,我们收集了一个Makeup Transfer In the Wild数据集,其中包含具有不同姿势和表情的图像,以及一个Makeup Transfer High-Resolution数据集,其中包含高分辨率图像。实验表明,PSGAN++不仅可以在大的姿势/表情差异的情况下实现最先进的精细化妆…