Re-identification of individual animals in images can be ambiguous due to subtle variations in body markings between different individuals and no constraints on the poses of animals in the wild. Person re-identification is a similar task and it has been approached with a deep convolutional neural network (CNN) that learns discriminative embeddings for images of people. However, learning discriminative features for an individual animal is more challenging than for a person's appearance due to the relatively small size of ecological datasets compared to labelled datasets of person's identities. We propose to improve embedding learning by exploiting body landmarks information explicitly. Body landmarks are provided to the input of a CNN as confidence heatmaps that can be obtained from a separate body landmark predictor. The model is encouraged to use heatmaps by learning an auxiliary task of reconstructing input heatmaps. Body landmarks guide a feature extraction network to learn the representation of a distinctive pattern and its position on the body. We evaluate the proposed method on a large synthetic dataset and a small real dataset. Our method outperforms the same model without body landmarks input by 26% and 18% on the synthetic and the real datasets respectively. The method is robust to noise in input coordinates and can tolerate an error in coordinates up to 10% of the image size.
LaCoVL-FER: Landmark-Guided Contrastive Learning Network with Vision-Language Enhancement for Facial Expression Recognition
School of Airspace and Engineering, Shandong University（山东大学航空航天与工程学院）；School of Computer Science and Technology, Shandong University of Finance and Economics（山东财经大学计算机科学与技术学院）；School of Psychology and Neuroscience, University of Glasgow（格拉斯哥大学心理学与神经科学学院）；Faculty of Information Science and Engineering, Ocean University of China（中国海洋大学信息科学与工程学院）
Queensland University of Technology（昆士兰科技大学）；The University of Queensland（昆士兰大学）
Knowledge about the locations of keypoints of an object in an image can assist in fine-grained classification and identification tasks, particularly for the case of objects that exhibit large variations in poses that greatly influence their visual appearance, such as wild animals. However, supervised training of a keypoint detection network requires annotating a large image dataset for each animal species, which is a labor-intensive task. To reduce the need for labeled data, we propose to learn simultaneously keypoint heatmaps and pose invariant keypoint representations in a semi-supervised ma…
PLGSA-Transformer: Periocular Landmark-Guided Attention with Occlusion-Adaptive Cosine Thresholding for Cross-Modal Masked and Unmasked Face Recognition
Faculty of Engineering and Computer Science, Qaiwan International University（工程与计算机科学学院，卡伊万国际大学）；Department of Information and Communication Technology Center (ICTC)-System Information, Ministry of Higher Education and Scientific Research（信息与通信技术中心（ICTC）-系统信息，高等教育与科学研究部）
Universidade Lisboa（里斯本大学）；Farfetch（发发奇）；Carnegie Mellon University（卡内基梅隆大学）
We propose a compact framework with guided attention for multi-label classification in the fashion domain. Our visual semantic attention model (VSAM) is supervised by automatic pose extraction creating a discriminative feature space. VSAM outperforms the state of the art for an in-house dataset and performs on par with previous works on the DeepFashion dataset, even without using any landmark annotations. Additionally, we show that our semantic attention module brings robustness to large quantities of wrong annotations and provides more interpretable results.
Boston University（波士顿大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Image and video analysis is often a crucial step in the study of animal behavior and kinematics. Often these analyses require that the position of one or more animal landmarks are annotated (marked) in numerous images. The process of annotating landmarks can require a significant amount of time and tedious labor, which motivates the need for algorithms that can automatically annotate landmarks. In the community of scientists that use image and video analysis to study the 3D flight of animals, there has been a trend of developing more automated approaches for annotating landmarks, yet they fall…
Recent technological developments have spurred great advances in the computerized tracking of joints and other landmarks in moving animals, including humans. Such tracking promises important advances in biology and biomedicine. Modern tracking models depend critically on labor-intensive annotated datasets of primary landmarks by non-expert humans. However, such annotation approaches can be costly and impractical for secondary landmarks, that is, ones that reflect fine-grained geometry of animals, and that are often specific to customized behavioral tasks. Due to visual and geometric ambiguity,…
Peking University（北京大学）；Deepwise AI Lab（深睿医疗人工智能实验室）；Microsoft Research Asia（微软亚洲研究院）；The University of Hong Kong（香港大学）
As an important upstream task for many medical applications, supervised landmark localization still requires non-negligible annotation costs to achieve desirable performance. Besides, due to cumbersome collection procedures, the limited size of medical landmark datasets impacts the effectiveness of large-scale self-supervised pre-training methods. To address these challenges, we propose a two-stage framework for one-shot medical landmark localization, which first infers landmarks by unsupervised registration from the labeled exemplar to unlabeled targets, and then utilizes these noisy pseudo l…
This paper proposes a CNN cascade for semantic part segmentation guided by pose-specific information encoded in terms of a set of landmarks (or keypoints). There is large amount of prior work on each of these tasks separately, yet, to the best of our knowledge, this is the first time in literature that the interplay between pose estimation and semantic part segmentation is investigated. To address this limitation of prior work, in this paper, we propose a CNN cascade of tasks that firstly performs landmark localisation and then uses this information as input for guiding semantic part segmentat…
University of Trento（特伦托大学）；EPFL（洛桑联邦理工学院）；Inria（法国国家信息与自动化研究所）；Grenoble-Alpes University（格勒诺布尔阿尔卑斯大学）；Fondazione Bruno Kessler (FBK)（布鲁诺·凯斯勒基金会）
Each smile is unique: one person surely smiles in different ways (e.g., closing/opening the eyes or mouth). Given one input image of a neutral face, can we generate multiple smile videos with distinctive characteristics? To tackle this one-to-many video generation problem, we propose a novel deep learning architecture named Conditional Multi-Mode Network (CMM-Net). To better encode the dynamics of facial expressions, CMM-Net explicitly exploits facial landmarks for generating smile sequences. Specifically, a variational auto-encoder is used to learn a facial landmark embedding. This single emb…
Tianjin University（天津大学）；Wuhan University（武汉大学）；Tencent AI Lab（腾讯人工智能实验室）；Stony Brook University（石溪大学）
It is challenging to inpaint face images in the wild, due to the large variation of appearance, such as different poses, expressions and occlusions. A good inpainting algorithm should guarantee the realism of output, including the topological structure among eyes, nose and mouth, as well as the attribute consistency on pose, gender, ethnicity, expression, etc. This paper studies an effective deep learning based strategy to deal with these issues, which comprises of a facial landmark predicting subnet and an image inpainting subnet. Concretely, given partial observation, the landmark predictor …
Structure-Guided Self-Supervised Matching for One-Shot Medical Landmark Detection
Department of Computer Science（计算机科学系）；Renmin University of China（中国人民大学）；University of Science and Technology of China（中国科学技术大学）；School of Biomedical Engineering（生物医学工程学院）；School of Artificial Intelligence（人工智能学院）；Shanghai Jiaotong University（上海交通大学）；Department of Orthopedics（骨科系）；Affiliated Hospital of Guizhou Medical University（贵州医科大学附属医院）；Yale University（耶鲁大学）
We propose a simple yet powerful Landmark guided Generative Adversarial Network (LandmarkGAN) for the facial expression-to-expression translation using a single image, which is an important and challenging task in computer vision since the expression-to-expression translation is a non-linear and non-aligned problem. Moreover, it requires a high-level semantic understanding between the input and output images since the objects in images can have arbitrary poses, sizes, locations, backgrounds, and self-occlusions. To tackle this problem, we propose utilizing facial landmark information explicitl…
Universidade Federal de Minas Gerais (UFMG)（米纳斯吉拉斯联邦大学（UFMG））
The detection of fiducial points on faces has significantly been favored by the rapid progress in the field of machine learning, in particular in the convolution networks. However, the accuracy of most of the detectors strongly depends on an enormous amount of annotated data. In this work, we present a domain adaptation approach based on a two-step learning to detect fiducial points on human and animal faces. We evaluate our method on three different datasets composed of different animal faces (cats, dogs, and horses). The experiments show that our method performs better than state of the art …
This paper addresses the problem of very large-scale image retrieval, focusing on improving its accuracy and robustness. We target enhanced robustness of search to factors such as variations in illumination, object appearance and scale, partial occlusions, and cluttered backgrounds - particularly important when search is performed across very large datasets with significant variability. We propose a novel CNN-based global descriptor, called REMAP, which learns and aggregates a hierarchy of deep features from multiple CNN layers, and is trained end-to-end with a triplet loss. REMAP explicitly l…
FGTBT: Frequency-Guided Task-Balancing Transformer for Unified Facial Landmark Detection
School of Information Engineering, Zhongnan University of Economics（经济法律大学信息工程学院）；School of Computer Science and Engineering, Nanyang Technological University, Singapore（南洋理工大学计算机科学与工程学院）；College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China（深圳大学计算机科学与软件工程学院）；School of Information Science and Engineering, Yunnan University, Kunming, Yunnan, China（云南大学信息科学与工程学院）
We present a novel method for multi image domain and multi-landmark definition learning for small dataset facial localization. Training a small dataset alongside a large(r) dataset helps with robust learning for the former, and provides a universal mechanism for facial landmark localization for new and/or smaller standard datasets. To this end, we propose a Vision Transformer encoder with a novel decoder with a definition agnostic shared landmark semantic group structured prior, that is learnt, as we train on more than one dataset concurrently. Due to our novel definition agnostic group prior …
Indian Statistical Institute（印度统计研究所）；Institute for Datability Science, Osaka University（大阪大学数据可用性科学研究所）
In this paper, we propose a Landmark Guided Virtual Try-On (LGVTON) method for clothes, which aims to solve the problem of clothing trials on e-commerce websites. Given the images of two people: a person and a model, it generates a rendition of the person wearing the clothes of the model. This is useful considering the fact that on most e-commerce websites images of only clothes are not usually available. We follow a three-stage approach to achieve our objective. In the first stage, LGVTON warps the clothes of the model using a Thin-Plate Spline (TPS) based transformation to fit the person. Un…
Pose-Guided Self-Training with Two-Stage Clustering for Unsupervised Landmark Discovery
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Heidelberg（海德堡大学）；Information Technology University of Punjab（旁遮普信息技术大学）
Nowadays, due to the ubiquitous visual media there are vast amounts of already available high-resolution (HR) face images. Therefore, for super-resolving a given very low-resolution (LR) face image of a person it is very likely to find another HR face image of the same person which can be used to guide the process. In this paper, we propose a convolutional neural network (CNN)-based solution, namely GWAInet, which applies super-resolution (SR) by a factor 8x on face images guided by another unconstrained HR face image of the same person with possible differences in age, expression, pose or siz…
Weight sharing has become a de facto standard in neural architecture search because it enables the search to be done on commodity hardware. However, recent works have empirically shown a ranking disorder between the performance of stand-alone architectures and that of the corresponding shared-weight networks. This violates the main assumption of weight-sharing NAS algorithms, thus limiting their effectiveness. We tackle this issue by proposing a regularization term that aims to maximize the correlation between the performance rankings of the shared-weight network and that of the standalone arc…
Generalizable Face Landmarking Guided by Conditional Face Warping
School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Key Laboratory of Artificial Intelligence, Ministry of Education, Shanghai（上海 教育部人工智能重点实验室）
Sheep Facial Pain Assessment Under Weighted Graph Neural Networks
CISTER Research Center, Porto, Portugal（CISTER研究中心，葡萄牙 Porto）；Faculty of Engineering University of Porto, Portugal（葡萄牙 Porto大学工程学院）；Univ. Lille, CNRS, Centrale Lille, UMR 9189 CRIStAL, F-59000, Lille, France（法国 Lille 大学，CNRS，Centrale Lille，UMR 9189 CRIStAL）；IMT Nord Europe, Institut Mines-Télécom, Univ. Lille, Centre for Digital Systems, F-59000 , Lille, France（法国 IMT Nord Europe，Institut Mines-Télécom，Univ. Lille，数字系统中心）
University of Vienna（维也纳大学）；NTNU Trondheim（挪威科技大学）
Motions of virtual characters in movies or video games are typically generated by recording actors using motion capturing methods. Animations generated this way often need postprocessing, such as improving the periodicity of cyclic animations or generating entirely new motions by interpolation of existing ones. Furthermore, search and classification of recorded motions becomes more and more important as the amount of recorded motion data grows. In this paper, we will apply methods from shape analysis to the processing of animations. More precisely, we will use the by now classical elastic metr…
University of Nottingham（诺丁汉大学）；Inception Institute of Artificial Intelligence（人工智能起源研究所）；Comsats University Islamabad（COMSATS伊斯兰堡大学）
Being heavily reliant on animals, it is our ethical obligation to improve their well-being by understanding their needs. Several studies show that animal needs are often expressed through their faces. Though remarkable progress has been made towards the automatic understanding of human faces, this has regrettably not been the case with animal faces. There exists significant room and appropriate need to develop automatic systems capable of interpreting animal faces. Among many transformative impacts, such a technology will foster better and cheaper animal healthcare, and further advance animal …
HieDG: A Hierarchical Discrete Geometry-Guided Framework for Multi-Animal Tracking
State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所脑认知与脑启发智能技术国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Technology, Beijing Forestry University（北京林业大学工学院）；School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学前沿交叉科学学院）；College of Computer Science, Sichuan University（四川大学计算机学院）；C²DL, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所C²DL）；School of Automation, Chongqing University（重庆大学自动化学院）；Faculty of Life and Health Sciences, Shenzhen University of Advanced Technology（深圳理工大学生命健康学院）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）
School of Electronic Engineering, Xidian University（西安电子科技大学电子工程学院）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Cephalometric tracing method is usually used in orthodontic diagnosis and treatment planning. In this paper, we propose a deep learning based framework to automatically detect anatomical landmarks in cephalometric X-ray images. We train the deep encoder-decoder for landmark detection, and combine global landmark configuration with local high-resolution feature responses. The proposed frame-work is based on 2-stage u-net, regressing the multi-channel heatmaps for land-mark detection. In this framework, we embed attention mechanism with global stage heatmaps, guiding the local stage inferring, t…
Fine-grained classification remains a challenging task because distinguishing categories needs learning complex and local differences. Diversity in the pose, scale, and position of objects in an image makes the problem even more difficult. Although the recent Vision Transformer models achieve high performance, they need an extensive volume of input data. To encounter this problem, we made the best use of GAN-based data augmentation to generate extra dataset instances. Oxford-IIIT Pets was our dataset of choice for this experiment. It consists of 37 breeds of cats and dogs with variations in sc…
UniMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation
The Hong Kong University of Science and Technology（香港科学与技术大学）；The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；Peking University（北京大学）；Tencent（腾讯）；National University of Singapore（新加坡国立大学）
SwinIFS: Landmark Guided Swin Transformer For Identity Preserving Face Super Resolution
King Fahd University of Petroleum and Minerals（国王法赫德石油和矿物大学）；The University of Western Australia（西澳大学）；The University of Canberra（堪培拉大学）；The University of Regina（里贾纳大学）
Scene graph generation (SGG) is a sophisticated task that suffers from both complex visual features and dataset long-tail problem. Recently, various unbiased strategies have been proposed by designing novel loss functions and data balancing strategies. Unfortunately, these unbiased methods fail to emphasize language priors in feature refinement perspective. Inspired by the fact that predicates are highly correlated with semantics hidden in subject-object pair and global context, we propose LANDMARK (LANguage-guiDed representationenhanceMent frAmewoRK) that learns predicate-relevant representat…
UCSD（加州大学圣地亚哥分校）；Salk Institute for Biological Studies（索尔克生物研究所）
Brightfield and fluorescent imaging of whole brain sections are funda- mental tools of research in mouse brain study. As sectioning and imaging become more efficient, there is an increasing need to automate the post-processing of sec- tions for alignment and three dimensional visualization. There is a further need to facilitate the development of a digital atlas, i.e. a brain-wide map annotated with cell type and tract tracing data, which would allow the automatic registra- tion of images stacks to a common coordinate system. Currently, registration of slices requires manual identification of …
Geometric-Guided Few-Shot Dental Landmark Detection with Human-Centric Foundation Model
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China（电子与计算机工程系，香港科学与技术大学）；Division of Paediatric Dentistry and Orthodontics, Faculty of Dentistry, The University of Hong Kong, Hong Kong SAR, China（牙科学院儿童牙科与正畸学系，香港大学）
Siemens Healthineers（西门子医疗）；Imperial College London（帝国理工学院）
Real-time localization of prostate gland in trans-rectal ultrasound images is a key technology that is required to automate the ultrasound guided prostate biopsy procedures. In this paper, we propose a new deep learning based approach which is aimed at localizing several prostate landmarks efficiently and robustly. We propose a multitask learning approach primarily to make the overall algorithm more contextually aware. In this approach, we not only consider the explicit learning of landmark locations, but also build-in a mechanism to learn the contour of the prostate. This multitask learning i…
G$^2$V$^2$former: Graph Guided Video Vision Transformer for Face Anti-Spoofing
University of Science and Technology of China（中国科学技术大学）；Great Bay University（大湾区大学）；Anhui Tsinglink Information Tech. Com Ltd.（安徽清link信息技术有限公司）；South China University of Technology（华南理工大学）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）
Fundus photography has been routinely used to document the presence and severity of various retinal degenerative diseases such as age-related macula degeneration, glaucoma, and diabetic retinopathy, for which the fovea, optic disc (OD), and optic cup (OC) are important anatomical landmarks. Identification of those anatomical landmarks is of great clinical importance. However, the presence of lesions, drusen, and other abnormalities during retinal degeneration severely complicates automatic landmark detection and segmentation. Most existing works treat the identification of each landmark as a s…
