Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation
Carnegie Mellon University(卡内基梅隆大学);Alibaba DAMO Academy(阿里巴巴达摩院);Fudan University(复旦大学);University of New South Wales(新南威尔士大学);Australian National University(澳大利亚国立大学);INF Technology(INF科技)
Junyan Wang, Zhenhong Sun, Zhiyu Tan, Xuanbai Chen, Weihua Chen, Hao Li, Cheng Zhang, Yang Song
Vanilla text-to-image diffusion models struggle with generating accurate human images, commonly resulting in imperfect anatomies such as unnatural postures or disproportionate limbs.Existing methods address this issue mostly by fine-tuning the model with extra images or adding additional controls -- human-centric priors such as pose or depth maps -- during the image generation phase. This paper explores the integration of these human-centric priors directly into the model fine-tuning stage, essentially eliminating the need for extra conditions at the inference stage. We realize this idea by proposing a human-centric alignment loss to strengthen human-related information from the textual prompts within the cross-attention maps. To ensure semantic detail richness and human structural accuracy during fine-tuning, we introduce scale-aware and step-wise constraints within the diffusion process, according to an in-depth analysis of the cross-attention layer. Extensive experiments show that our method largely improves over state-of-the-art text-to-image models to synthesize high-quality human images based on user-written prompts. Project page: \url{https://hcplayercvpr2024.github.io}.
MoLE: Enhancing Human-centric Text-to-image Diffusion via Mixture of Low-rank Experts
Key Lab of High Confidence Software Technologies (Peking University)(高可信软件技术重点实验室(北京大学));School of Computer Science, Peking University(北京大学计算机学院);Johns Hopkins University(约翰斯·霍普金斯大学);UC Berkeley(加州大学伯克利分校);The University of Hong Kong(香港大学);Baidu(百度)
Concept-Centric Token Interpretation for Vector-Quantized Generative Models
School of Computing, University of Georgia(佐治亚大学计算学院);Department of Data Science, New Jersey Institute of Technology(新泽西理工学院数据科学系);Department of Computer Science, New York University(纽约大学计算机科学系)
RobotEQ-Video: A Video-Centric Benchmark for Social Proactive Intelligence with World-State Taxonomy
State Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University(同济大学自主智能无人系统全国重点实验室);Tsinghua University(清华大学);The Chinese University of Hong Kong(香港中文大学);Shenzhen MSU-BIT University(深圳北理莫斯科大学);University of Science and Technology of China(中国科学技术大学)
HAP: Structure-Aware Masked Image Modeling for Human-Centric Perception
Zhejiang University(浙江大学);Baidu VIS(百度视觉技术部);University of Science and Technology Beijing(北京科技大学);South China University of Technology(华南理工大学);Shanghai Jiao Tong University(上海交通大学);Jilin University(吉林大学)
Omni-Scene: Omni-Gaussian Representation for Ego-Centric Sparse-View Scene Reconstruction
School of Engineering, Westlake University(西湖大学工学院);Institute of Advanced Technology, Westlake Institute for Advanced Study(西湖高等研究院先进技术研究院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)
3D Feature Distillation with Object-Centric Priors
Department of Artificial Intelligence University of Groningen, the Neteherlands(格罗宁根大学人工智能系);School of Informatics University of Edinburgh, United Kingdom(爱丁堡大学信息学院);Department of Computer Science University College London, United Kingdom(伦敦大学学院计算机科学系)
Primus: Enforcing Attention Usage for 3D Medical Image Segmentation
Division of Medical Image Computing, German Cancer Research Center, Heidelberg(德国癌症研究中心医学图像计算部门,海德堡);Faculty of Mathematics and Computer Science, Heidelberg University(海德堡大学数学与计算机科学学院);Helmholtz Imaging, German Cancer Research Center(德国癌症研究中心赫尔姆霍茨成像中心);Medical Faculty, Heidelberg University(海德堡大学医学系);National Center for Tumor Diseases (NCT), NCT Heidelberg(海德堡国家肿瘤疾病中心(NCT));Pattern Analysis and Learning Group, Department of Radiation Oncology(放射肿瘤学系模式分析与学习小组);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi(阿布扎克穆罕默德·本·扎耶德人工智能大学)