We introduce Omni-ID, a novel facial representation designed specifically for generative tasks. Omni-ID encodes holistic information about an individual's appearance across diverse expressions and poses within a fixed-size representation. It consolidates information from a varied number of unstructured input images into a structured representation, where each entry represents certain global or local identity features. Our approach uses a few-to-many identity reconstruction training paradigm, where a limited set of input images is used to reconstruct multiple target images of the same individual in various poses and expressions. A multi-decoder framework is further employed to leverage the complementary strengths of diverse decoders during training. Unlike conventional representations, such as CLIP and ArcFace, which are typically learned through discriminative or contrastive objectives, Omni-ID is optimized with a generative objective, resulting in a more comprehensive and nuanced identity capture for generative tasks. Trained on our MFHQ dataset -- a multi-view facial image collection, Omni-ID demonstrates substantial improvements over conventional representations across various generative tasks.
DreamVideo-Omni: Omni-Motion Controlled Multi-Subject Video Customization with Latent Identity Reinforcement Learning
Fudan University（复旦大学）；The Hong Kong University of Science and Technology（香港科技大学）；Tongyi Lab, Alibaba Group（阿里云实验室）；Zhejiang University（浙江大学）；MMLab, The University of Hong Kong（香港大学MMLab）；Nanyang Technological University（南洋理工大学）；Show Lab, National University of Singapore（新加坡国立大学Show Lab）
Omni-MMSI: Toward Identity-attributed Social Interaction Understanding
University of Texas at Dallas（德克萨斯大学达拉斯分校）；Georgia Institute of Technology（佐治亚理工学院）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding
The Hong Kong Polytechnic University（香港理工大学）；OPPO Research Institute（OPPO研究院）；Colorado School of Mines（科罗拉多矿业学院）；Nanyang Technological University（南洋理工大学）；Shanghai Jiao Tong University（上海交通大学）
Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmony
Wolf 1069B Lab, Sany Group（沃尔夫1069B实验室，三一集团）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of Engineering and Technology, University of Washington, Tacoma（华盛顿大学塔科姆分校工程与科技学院）；Alibaba Group（阿里巴巴集团）；Department of Engineering, King’s College London（伦敦大学国王学院工程系）；Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）
Object parsing -- the task of decomposing an object into its semantic parts -- has traditionally been formulated as a category-level segmentation problem. Consequently, when there are multiple objects in an image, current methods cannot count the number of objects in the scene, nor can they determine which part belongs to which object. We address this problem by segmenting the parts of objects at an instance-level, such that each pixel in the image is assigned a part label, as well as the identity of the object it belongs to. Moreover, we show how this approach benefits us in obtaining segment…
Xi’an Jiaotong University（西安交通大学）；Tsinghua University（清华大学）；DAMO Academy, Alibaba Group（阿里云实验室）；Chinese Academy of Sciences（中国科学院）；National University of Singapore（新加坡国立大学）
Advanced Acceptance Score: A Holistic Measure for Biometric Quantification
Dept. of Electrical Engineering, IIT Delhi, India（电子工程系，德里印度理工学院）；Dept. of Electrical Engineering and Bharti School of Telecommunications at IIT Delhi, India（电子工程系和Bharti电信学院，德里印度理工学院）
Beijing University of Posts and Telecommunications（北京邮电大学）
Facial expression recognition (FER) is a challenging problem because the expression component is always entangled with other irrelevant factors, such as identity and head pose. In this work, we propose an identity and pose disentangled facial expression recognition (IPD-FER) model to learn more discriminative feature representation. We regard the holistic facial representation as the combination of identity, pose and expression. These three components are encoded with different encoders. For identity encoder, a well pre-trained face recognition model is utilized and fixed during training, whic…
3D face modeling has been an active area of research in computer vision and computer graphics, fueling applications ranging from facial expression transfer in virtual avatars to synthetic data generation. Existing 3D deep learning generative models (e.g., VAE, GANs) allow generating compact face representations (both shape and texture) that can model non-linearities in the shape and appearance space (e.g., scatter effects, specularities, etc.). However, they lack the capability to control the generation of subtle expressions. This paper proposes a new 3D face generative model that can decouple…
Bridging Information Asymmetry: A Hierarchical Framework for Deterministic Blind Face Restoration
Biomedical Engineering Department, College of Future Technology, Peking University（北京大学未来技术学院生物医学工程系）；Institute of Medical Technology, Peking University Health Science Center（北京大学医学部医学技术研究所）；National Biomedical Imaging Center, Peking University（北京大学国家生物医学成像中心）
Certain facial parts are salient (unique) in appearance, which substantially contribute to the holistic recognition of a subject. Occlusion of these salient parts deteriorates the performance of face recognition algorithms. In this paper, we propose a generative model to reconstruct the missing parts of the face which are under occlusion. The proposed generative model (SD-GAN) reconstructs a face preserving the illumination variation and identity of the face. A novel adversarial training algorithm has been designed for a bimodal mutually exclusive Generative Adversarial Network (GAN) model, fo…
Generative adversarial networks (GANs) learn to synthesise new samples from a high-dimensional distribution by passing samples drawn from a latent space through a generative network. When the high-dimensional distribution describes images of a particular data set, the network should learn to generate visually similar image samples for latent variables that are close to each other in the latent space. For tasks such as image retrieval and image classification, it may be useful to exploit the arrangement of the latent space by projecting images into it, and using this as a representation for dis…
