In this paper we present a new data-driven method for robust skin detection from a single human portrait image. Unlike previous methods, we incorporate human body as a weak semantic guidance into this task, considering acquiring large-scale of human labeled skin data is commonly expensive and time-consuming. To be specific, we propose a dual-task neural network for joint detection of skin and body via a semi-supervised learning strategy. The dual-task network contains a shared encoder but two decoders for skin and body separately. For each decoder, its output also serves as a guidance for its counterpart, making both decoders mutually guided. Extensive experiments were conducted to demonstrate the effectiveness of our network with mutual guidance, and experimental results show our network outperforms the state-of-the-art in skin detection.
Human skin segmentation is a crucial task in computer vision and biometric systems, yet it poses several challenges such as variability in skin color, pose, and illumination. This paper presents a robust data-driven skin segmentation method for a single image that addresses these challenges through the integration of contextual information and efficient network design. In addition to robustness and accuracy, the integration into real-time systems requires a careful balance between computational power, speed, and performance. The proposed method incorporates two attention modules, Body Attentio…
Northwestern Polytechnical University（西北工业大学）；University of Adelaide（阿德莱德大学）
Automated skin lesion segmentation and classification are two most essential and related tasks in the computer-aided diagnosis of skin cancer. Despite their prevalence, deep learning models are usually designed for only one task, ignoring the potential benefits in jointly performing both tasks. In this paper, we propose the mutual bootstrapping deep convolutional neural networks (MB-DCNN) model for simultaneous skin lesion segmentation and classification. This model consists of a coarse segmentation network (coarse-SN), a mask-guided classification network (mask-CN), and an enhanced segmentati…
SkinMamba: A Precision Skin Lesion Segmentation Architecture with Cross-Scale Global State Modeling and Frequency Boundary Guidance
College of Artificial Intelligence, Nanjing Agricultural University（南京农业大学人工智能学院）；State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）
Tianjin University（天津大学）；University of South Carolina（南卡罗来纳大学）
Referring expression comprehension (REC) aims to localize a text-related region in a given image by a referring expression in natural language. Existing methods focus on how to build convincing visual and language representations independently, which may significantly isolate visual and language information. In this paper, we argue that for REC the referring expression and the target region are semantically correlated and subject, location and relationship consistency exist between vision and language.On top of this, we propose a novel approach called MutAtt to construct mutual guidance betwee…
The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；SmartMore Corporation（思谋科技）；The Chinese University of Hong Kong（香港中文大学）
Previous studies show the necessity of global and local adjustment for image enhancement. However, existing convolutional neural networks (CNNs) and transformer-based models face great challenges in balancing the computational efficiency and effectiveness of global-local information usage. Especially, existing methods typically adopt the global-to-local fusion mode, ignoring the importance of bidirectional interactions. To address those issues, we propose a novel mutual guidance network (MGN) to perform effective bidirectional global-local information exchange while keeping a compact architect…
Most deep learning object detectors are based on the anchor mechanism and resort to the Intersection over Union (IoU) between predefined anchor boxes and ground truth boxes to evaluate the matching quality between anchors and objects. In this paper, we question this use of IoU and propose a new anchor matching criterion guided, during the training phase, by the optimization of both the localization and the classification tasks: the predictions related to one task are used to dynamically assign sample anchors and improve the model on the other task, and vice versa. Despite the simplicity of the…
This paper introduces a new matting task called human instance matting (HIM), which requires the pertinent model to automatically predict a precise alpha matte for each human instance. Straightforward combination of closely related techniques, namely, instance segmentation, soft segmentation and human/conventional matting, will easily fail in complex cases requiring disentangling mingled colors belonging to multiple instances along hairy and thin boundary structures. To tackle these technical challenges, we propose a human instance matting framework, called InstMatt, where a novel mutual guida…
One Model for Two Tasks: Cooperatively Recognizing and Recovering Low-Resolution Scene Text Images by Iterative Mutual Guidance
School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Shanghai Key Lab of Intelligent Information Processing（上海智能信息处理重点实验室）；Tongji University（同济大学）
Department of Electrical Engineering, City Univeristy of Hong Kong, Hong Kong SAR, China
Automatic melanoma segmentation in dermoscopic images is essential in computer-aided diagnosis of skin cancer. Existing methods may suffer from the hole and shrink problems with limited segmentation performance. To tackle these issues, we propose a novel complementary network with adaptive receptive filed learning. Instead of regarding the segmentation task independently, we introduce a foreground network to detect melanoma lesions and a background network to mask non-melanoma regions. Moreover, we propose adaptive atrous convolution (AAC) and knowledge aggregation module (KAM) to fill holes a…
CNNs, Transformers, Hybrid, and Vision Language Models for Skin Cancer Detection
Department of Computer Science and Software Engineering, Concordia University, Montreal, Canada（计算机科学与软件工程系，康科迪亚大学，加拿大蒙特利尔）；Ebovir Biotechnologie Inc., Montreal, Canada（Ebovir生物技术公司，加拿大蒙特利尔）
USL-Net: Uncertainty Self-Learning Network for Unsupervised Skin Lesion Segmentation
School of Computing and Artificial Intelligence, Southwest Jiaotong University（西南交通大学计算机与人工智能学院）；Engineering Research Center of Sustainable Urban Intelligent Transportation, Ministry of Education（教育部可持续城市智能交通工程研究中心）；National Engineering Laboratory of Integrated Transportation Big Data Application Technology, Southwest Jiaotong University（西南交通大学综合交通大数据应用技术国家工程实验室）；Manufacturing Industry Chains Collaboration and Information Support Technology Key Laboratory of Sichuan Province, Southwest Jiaotong University（西南交通大学四川省制造业产业链协同与信息化支撑技术重点实验室）；Data Recovery Key Laboratory of Sichuan Province, Neijiang Normal University（内江师范学院四川省数据恢复重点实验室）
Early Unsupervised Domain Adaptation (UDA) methods have mostly assumed the setting of a single source domain, where all the labeled source data come from the same distribution. However, in practice the labeled data can come from multiple source domains with different distributions. In such scenarios, the single source domain adaptation methods can fail due to the existence of domain shifts across different source domains and multi-source domain adaptation methods need to be designed. In this paper, we propose a novel multi-source domain adaptation method, Mutual Learning Network for Multiple S…
Self-supervised monocular depth estimation presents a powerful method to obtain 3D scene information from single camera images, which is trainable on arbitrary image sequences without requiring depth labels, e.g., from a LiDAR sensor. In this work we present a new self-supervised semantically-guided depth estimation (SGDepth) method to deal with moving dynamic-class (DC) objects, such as moving cars and pedestrians, which violate the static-world assumptions typically made during training of such models. Specifically, we propose (i) mutually beneficial cross-domain training of (supervised) sem…
To eliminate the problems of large dimensional differences, big semantic gap, and mutual interference caused by hybrid features, in this paper, we propose a novel Multi-Features Guidance Network for partial-to-partial point cloud registration(MFG). The proposed network mainly includes four parts: keypoints' feature extraction, correspondences searching, correspondences credibility computation, and SVD, among which correspondences searching and correspondence credibility computation are the cores of the network. Unlike the previous work, we utilize the shape features and the spatial coordinates…
DiffMIC: Dual-Guidance Diffusion Network for Medical Image Classification
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Institute of High Performance Computing, Agency for Science, Technology and Research（新加坡科技研究局高性能计算研究所）；University of Cambridge（剑桥大学）
Inception Institute of Artificial Intelligence（起源人工智能研究院）；Northwestern Polytechnical University（西北工业大学）
The application of light field data in salient object de-tection is becoming increasingly popular recently. The diffi-culty lies in how to effectively fuse the features within the fo-cal stack and how to cooperate them with the feature of theall-focus image. Previous methods usually fuse focal stackfeatures via convolution or ConvLSTM, which are both lesseffective and ill-posed. In this paper, we model the infor-mation fusion within focal stack via graph networks. Theyintroduce powerful context propagation from neighbouringnodes and also avoid ill-posed implementations. On the onehand, we cons…
RSONet: Region-guided Selective Optimization Network for RGB-T Salient Object Detection
School of Control Science and Engineering, Shandong University（控制科学与工程学院，山东大学）；Key Laboratory of Machine Intelligence and System Control, Ministry of Education（机器智能与系统控制重点实验室，教育部）；School of Automation, Hangzhou Dianzi University（自动化学院，杭州电子科技大学）；School of information engineering, Huzhou University（信息工程学院，湖州大学）；School of Data Science, Lingnan University（数据科学学院，岭南大学）
Ensemble Deep Learning and LLM-Assisted Reporting for Automated Skin Lesion Diagnosis
Institute of Computer Science The University of Agriculture Peshawar, Pakistan（计算机科学研究所 伊斯兰农业大学 薩瓦爾，巴基斯坦）；Department of Computer Science Islamia College Peshawar Pakistan（计算机科学系 伊斯兰学院 薩瓦爾，巴基斯坦）；Department of Software Engineering Islamia College Peshawar Pakistan（软件工程系 伊斯兰学院 薩瓦爾，巴基斯坦）
Salient Object Detection is the task of predicting the human attended region in a given scene. Fusing depth information has been proven effective in this task. The main challenge of this problem is how to aggregate the complementary information from RGB modality and depth modality. However, conventional deep models heavily rely on CNN feature extractors, and the long-range contextual dependencies are usually ignored. In this work, we propose Dual Swin-Transformer based Mutual Interactive Network. We adopt Swin-Transformer as the feature extractor for both RGB and depth modality to model the lo…
SocialMirror: Reconstructing 3D Human Interaction Behaviors from Monocular Videos with Semantic and Geometric Guidance
ShanghaiTech University（上海科技大学）；Nanyang Technological University（南洋理工大学）；Guangzhou Institute of Energy Conversion, CAS（广州能源研究所，中国科学院）；University of Science and Technology of China（中国科学技术大学）；The Chinese University of Hong Kong（香港中文大学）；Inceptio Technology（Inceptio科技）；Shanghai Jiao Tong University（上海交通大学）
Learning Where to Focus: Density-Driven Guidance for Detecting Dense Tiny Objects
Key Laboratory of Intelligent Computing & Signal Processing (Anhui University), Ministry of Education（智能计算与信号处理重点实验室（安徽大学）、教育部）；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；th Research Institute, China Electronics Technology Group Corporation（中国电子科技集团第三十八研究所）；Collaborative Innovation Laboratory for Computer Vision and Pattern Recognition, School of Artificial Intelligence and Big Data, Hefei University（计算机视觉与模式识别协同创新实验室，合肥大学人工智能与大数据学院）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）
MuNet: A Mutualistic Network for Joint 3D Human Mesh Recovery and 3D Clothed Human Reconstruction from Single Images
National Engineering Research Center for E-Learning（教育信息化国家级工程研究中心）；National Engineering Research Center of Educational Big Data（教育大数据国家级工程研究中心）；School of Electronic Information and Communications（电子信息与通讯学院）；School of Artificial Intelligence and Automation（人工智能与自动化学院）
We propose a novel and unified Cycle in Cycle Generative Adversarial Network (C2GAN) for generating human faces, hands, bodies, and natural scenes. Our proposed C2GAN is a cross-modal model exploring the joint exploitation of the input image data and guidance data in an interactive manner. C2GAN contains two different generators, i.e., an image-generation generator and a guidance-generation generator. Both generators are mutually connected and trained in an end-to-end fashion and explicitly form three cycled subnets, i.e., one image generation cycle and two guidance generation cycles. Each cyc…
Centre Inria d’Université Côte d’Azur（蔚蓝海岸大学法国国家信息与自动化研究所中心）
Cross-modality data translation has attracted great interest in image computing. Deep generative models (\textit{e.g.}, GANs) show performance improvement in tackling those problems. Nevertheless, as a fundamental challenge in image translation, the problem of Zero-shot-Learning Cross-Modality Data Translation with fidelity remains unanswered. This paper proposes a new unsupervised zero-shot-learning method named Mutual Information guided Diffusion cross-modality data translation Model (MIDiffusion), which learns to translate the unseen source data to the target domain. The MIDiffusion leverag…
Sun Yat-sen University（中山大学）；ByteDance Inc.（字节跳动公司）；Baidu Inc.（百度公司）
In this paper, we introduce a novel task, referred to as Weakly-Supervised Spatio-Temporal Anomaly Detection (WSSTAD) in surveillance video. Specifically, given an untrimmed video, WSSTAD aims to localize a spatio-temporal tube (i.e., a sequence of bounding boxes at consecutive times) that encloses the abnormal event, with only coarse video-level annotations as supervision during training. To address this challenging task, we propose a dual-branch network which takes as input the proposals with multi-granularities in both spatial-temporal domains. Each branch employs a relationship reasoning m…
S$^2$ME: Spatial-Spectral Mutual Teaching and Ensemble Learning for Scribble-supervised Polyp Segmentation
The Chinese University of Hong Kong（香港中文大学）；National University of Singapore（新加坡国立大学）；Hubei University of Technology（湖北工业大学）；University College London（伦敦大学学院）
