Decoupled Cross-Modal Alignment Network for Text-RGBT Person Retrieval and A High-Quality Benchmark
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；National Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology（光电信息采集与防护技术国家重点实验室）；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）
Yifei Deng, Chenglong Li, Zhenyu Chen, Zihen Xu, Jin Tang
The performance of traditional text-image person retrieval task is easily affected by lighting variations due to imaging limitations of visible spectrum sensors. In recent years, cross-modal information fusion has emerged as an effective strategy to enhance retrieval robustness. By integrating complementary information from different spectral modalities, it becomes possible to achieve more stable person recognition and matching under complex real-world conditions. Motivated by this, we introduce a novel task: Text-RGBT Person Retrieval, which incorporates cross-spectrum information fusion by combining the complementary cues from visible and thermal modalities for robust person retrieval in challenging environments. The key challenge of Text-RGBT person retrieval lies in aligning text with multi-modal visual features. However, the inherent heterogeneity between visible and thermal modalities may interfere with the alignment between vision and language. To handle this problem, we propose a Decoupled Cross-modal Alignment network (DCAlign), which sufficiently mines the relationships between modality-specific and modality-collaborative visual with the text, for Text-RGBT person retrieval. To promote the research and development of this field, we create a high-quality Text-RGBT person retrieval dataset, RGBT-PEDES. RGBT-PEDES contains 1,822 identities from different age groups and genders with 4,723 pairs of calibrated RGB and T images, and covers high-diverse scenes from both daytime and nighttime with a various of challenges such as occlusion, weak alignment and adverse lighting conditions. Additionally, we carefully annotate 7,987 fine-grained textual descriptions for all RGBT person image pairs. Extensive experiments on RGBT-PEDES demonstrate that our method outperforms existing text-image person retrieval methods.
pFedSAM: Personalized Federated Learning of Segment Anything Model for Medical Image Segmentation
College of Computer Science and Technology, Zhejiang University, China（浙江大学计算机科学与技术学院）；Chinese Academy of Medical Sciences and Peking Union Medical College, China（中国医学科学院与北京协和医学院）；School of Computer Science, Wuhan University, China（武汉大学计算机学院）；Innovation Centre for Information, Binjiang Institute of Zhejiang University（浙江大学滨江学院信息创新中心）；Department of Data and Information, Children's Hospital, Zhejiang University School of Medicine（浙江大学医学院附属儿童医院数据与信息部）
Person image synthesis, e.g., pose transfer, is a challenging problem due to large variation and occlusion. Existing methods have difficulties predicting reasonable invisible regions and fail to decouple the shape and style of clothing, which limits their applications on person image editing. In this paper, we propose PISE, a novel two-stage generative model for Person Image Synthesis and Editing, which is able to generate realistic person images with desired poses, textures, or semantic layouts. For human pose transfer, we first synthesize a human parsing map aligned with the target pose to r…
View-decoupled Transformer for Person Re-identification under Aerial-ground Camera Network
School of Computer Science and Engineering, Sun Yat-Sen University（中山大学计算机科学与工程学院）；Department of Electrical and Computer Engineering, Johns Hopkins University（约翰霍普金斯大学电气与计算机工程系）；Pazhou Lab (HuangPu)（琶洲实验室（黄埔））；Guangdong Province Key Laboratory of Information Security Technology（广东省信息安全技术重点实验室）；Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education（机器智能与先进计算教育部重点实验室）
Huazhong University of Science and Technology（华中科技大学）；University of Technology Sydney（悉尼科技大学）；Baidu Research（百度研究院）
The goal of person search is to localize and match query persons from scene images. For high efficiency, one-step methods have been developed to jointly handle the pedestrian detection and identification sub-tasks using a single network. There are two major challenges in the current one-step approaches. One is the mutual interference between the optimization objectives of multiple sub-tasks. The other is the sub-optimal identification feature learning caused by small batch size when end-to-end training. To overcome these problems, we propose a decoupled and memory-reinforced network (DMRNet). …
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Idiap Research Institute（Idiap研究所）
We propose to leverage recent advances in reliable 2D pose estimation with Convolutional Neural Networks (CNN) to estimate the 3D pose of people from depth images in multi-person Human-Robot Interaction (HRI) scenarios. Our method is based on the observation that using the depth information to obtain 3D lifted points from 2D body landmark detections provides a rough estimate of the true 3D human pose, thus requiring only a refinement step. In that line our contributions are threefold. (i) we propose to perform 3D pose estimation from depth images by decoupling 2D pose estimation and 3D pose re…
MIT（麻省理工学院）；Facebook AI Research（脸书人工智能研究院）；CMU（卡内基梅隆大学）
We study a generalized setup for learning from demonstration to build an agent that can manipulate novel objects in unseen scenarios by looking at only a single video of human demonstration from a third-person perspective. To accomplish this goal, our agent should not only learn to understand the intent of the demonstrated third-person video in its context but also perform the intended task in its environment configuration. Our central insight is to enforce this structure explicitly during learning by decoupling what to achieve (intended task) from how to perform it (controller). We propose a …
Unlike prevalent facial expressions, micro expressions have subtle, involuntary muscle movements which are short-lived in nature. These minute muscle movements reflect true emotions of a person. Due to the short duration and low intensity, these micro-expressions are very difficult to perceive and interpret correctly. In this paper, we propose the dynamic representation of micro-expressions to preserve facial movement information of a video in a single frame. We also propose a Lateral Accretive Hybrid Network (LEARNet) to capture micro-level features of an expression in the facial region. The …
In person re-identification (ReID) tasks, many works explore the learning of part features to improve the performance over global image features. Existing methods explicitly extract part features by either using a hand-designed image division or keypoints obtained with external visual systems. In this work, we propose to learn Discriminative implicit Parts (DiPs) which are decoupled from explicit body parts. Therefore, DiPs can learn to extract any discriminative features that can benefit in distinguishing identities, which is beyond predefined body parts (such as accessories). Moreover, we pr…
D^3-Talker: Dual-Branch Decoupled Deformation Fields for Few-Shot 3D Talking Head Synthesis
School of Computer Science（计算机科学学院）；Software Engineering, Shenzhen University（软件工程，深圳大学）；Computer Vision Institute, School of Artificial Intelligence, Shenzhen University, Shenzhen, China（计算机视觉研究院，人工智能学院，深圳大学，深圳，中国）；Guangdong Provincial Key Laboratory of Intelligent Information Processing（广东省智能信息处理重点实验室）；HBUG Lab, University of Exeter（HBUG实验室，埃克塞特大学）
Tsinghua University（清华大学）；Addx Technology（Addx科技）；University of Science and Technology Beijing（北京科技大学）
Visual attention has proven to be effective in improving the performance of person re-identification. Most existing methods apply visual attention heuristically by learning an additional attention map to re-weight the feature maps for person re-identification. However, this kind of methods inevitably increase the model complexity and inference time. In this paper, we propose to incorporate the attention learning as additional objectives in a person ReID network without changing the original structure, thus maintain the same inference time and model size. Two kinds of attentions have been consi…
Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）；University of Science and Technology of China（中国科学技术大学）；Shanghai International Studies University（上海外国语大学）
Pose Guided Human Image Synthesis (PGHIS) is a challenging task of transforming a human image from the reference pose to a target pose while preserving its style. Most existing methods encode the texture of the whole reference human image into a latent space, and then utilize a decoder to synthesize the image texture of the target pose. However, it is difficult to recover the detailed texture of the whole human image. To alleviate this problem, we propose a method by decoupling the human body into several parts (\eg, hair, face, hands, feet, \etc) and then using each of these parts to guide th…
MIBE: Multi-subject Interaction Benchmark and Evaluator for Personalized Image Generation
University of California, Los Angeles（加州大学洛杉矶分校）；University of Southern California（南加州大学）；DeerLab LLC（DeerLab有限责任公司）；Carnegie Mellon University（卡内基梅隆大学）；Clemson University（克莱姆森大学）；Google（谷歌）；San Jose State University（圣何塞州立大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
