We present Score-Guided Human Mesh Recovery (ScoreHMR), an approach for solving inverse problems for 3D human pose and shape reconstruction. These inverse problems involve fitting a human body model to image observations, traditionally solved through optimization techniques. ScoreHMR mimics model fitting approaches, but alignment with the image observation is achieved through score guidance in the latent space of a diffusion model. The diffusion model is trained to capture the conditional distribution of the human model parameters given an input image. By guiding its denoising process with a task-specific score, ScoreHMR effectively solves inverse problems for various applications without the need for retraining the task-agnostic diffusion model. We evaluate our approach on three settings/applications. These are: (i) single-frame model fitting; (ii) reconstruction from multiple uncalibrated views; (iii) reconstructing humans in video sequences. ScoreHMR consistently outperforms all optimization baselines on popular benchmarks across all settings. We make our code and models available at the https://statho.github.io/ScoreHMR.
School of Communication Engineering, Hangzhou Dianzi University（杭州电子科技大学通信工程学院）
We present to recover the complete 3D facial geometry from a single depth view by proposing an Attention Guided Generative Adversarial Networks (AGGAN). In contrast to existing work which normally requires two or more depth views to recover a full 3D facial geometry, the proposed AGGAN is able to generate a dense 3D voxel grid of the face from a single unconstrained depth view. Specifically, AGGAN encodes the 3D facial geometry within a voxel space and utilizes an attention-guided GAN to model the illposed 2.5D depth-3D mapping. Multiple loss functions, which enforce the 3D facial geometry con…
AnimatableDreamer: Text-Guided Non-rigid 3D Model Generation and Reconstruction with Canonical Score Distillation
Tongji University（同济大学）；Tsinghua University（清华大学）；ShengShu（商汤科技（注：此处为机构音译，原文为ShengShu））；Xi’an Research Institute of High-Tech（西安高技术研究所）；Fudan University（复旦大学）；Zhejiang University（浙江大学）；Didi（滴滴出行）
MotionFollower: Editing Video Motion via Lightweight Score-Guided Diffusion
Fudan University（复旦大学）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）；Microsoft Research Asia（微软亚洲研究院）；Carnegie Mellon University（卡内基梅隆大学）；Huya Inc.（虎牙公司）
College of Engineering（工程学院）；Korea University（高丽大学）；School of Electrical Engineering（电气工程学院）
We propose a Generative Adversarial Network (GAN) that introduces an evaluator module using pre-trained networks. The proposed model, called score-guided GAN (ScoreGAN), is trained with an evaluation metric for GANs, i.e., the Inception score, as a rough guide for the training of the generator. By using another pre-trained network instead of the Inception network, ScoreGAN circumvents the overfitting of the Inception network in order that generated samples do not correspond to adversarial examples of the Inception network. Also, to prevent the overfitting, the evaluation metrics are employed o…
White Matter Geometry-Guided Score-Based Diffusion Model for Tissue Microstructure Imputation in Tractography Imaging
Harvard Medical School（哈佛医学院）；School of Computer Science, The University of Sydney（悉尼大学计算机学院）；Brigham and Women’s Hospital（布里格姆妇女医院）；University of Electronic Science and Technology of China（电子科技大学）
FDS: Frequency-Aware Denoising Score for Text-Guided Latent Diffusion Image Editing
School of Computer and Communication Sciences, EPFL（EPFL计算机与通信科学学院）；Department of Electrical and Photonics Engineering, DTU（丹麦技术大学电气与光子工程系）；Department of Electrical Engineering, Chalmers University of Technology（查尔姆斯理工大学电气工程系）
Department of Artificial Intelligence（人工智能系）；Kyungpook National University（庆尚国立大学）；Department of Electrical and Computer Engineering（电气电子工程系）；Queen’s University（皇后大学）；Department of Information and Communications Engineering（信息与通信工程系）；Pukyong National University（浦项国立大学）
Image super-resolution(SR) is fundamental to many vision system-from surveillance and autonomy to document analysis and retail analytics-because recovering high-frequency details, especially scene-text, enables reliable downstream perception. Scene-text, i.e., text embedded in natural images such as signs, product labels, and storefronts, often carries the most actionable information; when characters are blurred or hallucinated, optical character recognition(OCR) and subsequent decisions fail even if the rest of the image appears sharp. Yet previous SR research has often been tuned to distorti…
RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling
New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别新实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；The Hong Kong University of Science and Technology（香港科技大学）；Tencent（腾讯）
Text-guided image generation has progressed rapidly in recent years, inspiring major breakthroughs in text-guided shape generation. Recently, it has been shown that using score distillation, one can successfully text-guide a NeRF model to generate a 3D object. We adapt the score distillation to the publicly available, and computationally efficient, Latent Diffusion Models, which apply the entire diffusion process in a compact latent space of a pretrained autoencoder. As NeRFs operate in image space, a naive solution for guiding them with latent score distillation would require encoding to the …
Mitigating Exposure Bias in Discriminator Guided Diffusion Models
Artificial Intelligence and Learning Systems Laboratory（人工智能与学习系统实验室）；School of Electrical and Computer Engineering（电气与计算机工程学院）；National Technical University of Athens（雅典国立技术大学）
Class-frequency Guided Noise Schedule for Diffusion Models
Hefei University of Technology（合肥工业大学）；University of Science and Technology of China（中国科学技术大学）；The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
Wuhan University（武汉大学）；ByteDance AI Lab（字节跳动人工智能实验室）；NC State University（北卡罗来纳州立大学）
This paper presents a neural network built upon Transformers, namely PlaneTR, to simultaneously detect and reconstruct planes from a single image. Different from previous methods, PlaneTR jointly leverages the context information and the geometric structures in a sequence-to-sequence way to holistically detect plane instances in one forward pass. Specifically, we represent the geometric structures as line segments and conduct the network with three main components: (i) context and line segments encoders, (ii) a structure-guided plane decoder, (iii) a pixel-wise plane embedding decoder. Given a…
Automatic and accurate lesion segmentation is critical for clinically estimating the lesion statuses of stroke diseases and developing appropriate diagnostic systems. Although existing methods have achieved remarkable results, further adoption of the models is hindered by: (1) inter-class indistinction, the normal brain tissue resembles the lesion in appearance. (2) intra-class inconsistency, large variability exists between different areas of the lesion. To solve these challenges in stroke segmentation, we propose a novel method, namely Attention Guided Multiscale Recovery framework (AGMR-Net…
PixelFade: Privacy-preserving Person Re-identification with Noise-guided Progressive Replacement
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education（教育部机器智能与先进计算重点实验室）；Guangdong Province Key Laboratory of Information Security Technology（广东省信息安全技术重点实验室）
The Chinese University of Hong Kong（香港中文大学）；Sensetime Group Limited（商汤集团有限公司）
Monocular 3D object detection task aims to predict the 3D bounding boxes of objects based on monocular RGB images. Since the location recovery in 3D space is quite difficult on account of absence of depth information, this paper proposes a novel unified framework which decomposes the detection problem into a structured polygon prediction task and a depth recovery task. Different from the widely studied 2D bounding boxes, the proposed novel structured polygon in the 2D image consists of several projected surfaces of the target object. Compared to the widely-used 3D bounding box proposals, it is…
National Key Laboratory of Science and Technology on Multispectral Information Processing（多光谱信息处理科学与技术国家重点实验室）；Huazhong University of Science and Technology（华中科技大学）
Mismatch removal is a critical prerequisite in many feature-based tasks. Recent attempts cast the mismatch removal task as a binary classification problem and solve it through deep learning based methods. In these methods, the imbalance between positive and negative classes is important, which affects network performance, i.e., Fn-score. To establish the link between Fn-score and loss, we propose to guide the loss with the Fn-score directly. We theoretically demonstrate the direct link between our Guided Loss and Fn-score during training. Moreover, we discover that outliers often impair global…
MvKSR: Multi-view Knowledge-guided Scene Recovery for Hazy and Rainy Degradation
The Hong Kong Polytechnic University（香港理工大学）；Wuhan University of Technology（武汉理工大学）；Wuhan Baosight Software Co., Ltd（武汉宝信软件有限公司）；Singapore Management University（新加坡管理大学）
Score-Based Turbo Message Passing for Plug-and-Play Compressive Image Recovery
Department of Information Engineering, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong SAR（信息工程系，香港中文大学（深圳校区））；National Key Lab. of Wireless Commun., Uni. of Electronic Sci. and Tech. of China, Chengdu, China（无线通信国家重点实验室，电子科技大学，成都，中国）
Indian Institute of Technology, Bhubaneswar（布巴内斯瓦尔印度理工学院）；Artificial Intelligence and Robotics Institute, Korea Institute of Science and Technology（韩国科学技术院人工智能与机器人研究所）；Yonsei-KIST Convergence Research Institute, Yonsei University（延世大学延世-KIST融合研究院）
Unsupervised approaches for video anomaly detection may not perform as good as supervised approaches. However, learning unknown types of anomalies using an unsupervised approach is more practical than a supervised approach as annotation is an extra burden. In this paper, we use isolation tree-based unsupervised clustering to partition the deep feature space of the video segments. The RGB- stream generates a pseudo anomaly score and the flow stream generates a pseudo dynamicity score of a video segment. These scores are then fused using a majority voting scheme to generate preliminary bags of p…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；DiDi Labs（滴滴实验室）
We propose a novel approach, MUSE, to illustrate textual attributes visually via portrait generation. MUSE takes a set of attributes written in text, in addition to facial features extracted from a photo of the subject as input. We propose 11 attribute types to represent inspirations from a subject's profile, emotion, story, and environment. We propose a novel stacked neural network architecture by extending an image-to-image generative model to accept textual attributes. Experiments show that our approach significantly outperforms several state-of-the-art methods without using textual attribu…
Bielefeld University（比勒费尔德大学）；Queensland University of Technology (QUT)（昆士兰科技大学）
This work discusses a learning approach to mask rewarding objects in images using sparse reward signals from an imitation learning dataset. For that, we train an Hourglass network using only feedback from a critic model. The Hourglass network learns to produce a mask to decrease the critic's score of a high score image and increase the critic's score of a low score image by swapping the masked areas between these two images. We trained the model on an imitation learning dataset from the NeurIPS 2020 MineRL Competition Track, where our model learned to mask rewarding objects in a complex intera…
Predicting Multiple Clinical Outcomes Related to Functional Recovery and Social Isolation Among Older Adults After Lower-Limb Fracture or Hip Replacement
Liwa University（利瓦大学）；University of Toronto（多伦多大学）；University Health Network（大学健康网络）；United Arab Emirates University（阿联酋大学）；American University of the Middle East（中东美国大学）
