Current AI-assisted skin image diagnosis has achieved dermatologist-level performance in classifying skin cancer, driven by rapid advancements in deep learning architectures. However, unlike traditional vision tasks, skin images in general present unique challenges due to the limited availability of well-annotated datasets, complex variations in conditions, and the necessity for detailed interpretations to ensure patient safety. Previous segmentation methods have sought to reduce image noise and enhance diagnostic performance, but these techniques require fine-grained, pixel-level ground truth masks for training. In contrast, with the rise of foundation models, the Segment Anything Model (SAM) has been introduced to facilitate promptable segmentation, enabling the automation of the segmentation process with simple yet effective prompts. Efforts applying SAM predominantly focus on dermatoscopy images, which present more easily identifiable lesion boundaries than clinical photos taken with smartphones. This limitation constrains the practicality of these approaches to real-world applications. To overcome the challenges posed by noisy clinical photos acquired via non-standardized protocols and to improve diagnostic accessibility, we propose a novel Cross-Attentive Fusion framework for interpretable skin lesion diagnosis. Our method leverages SAM to generate visual concepts for skin diseases using prompts, integrating local visual concepts with global image features to enhance model performance. Extensive evaluation on two skin disease datasets demonstrates our proposed method's effectiveness on lesion diagnosis and interpretability.
A Novel Multi-Task Model Imitating Dermatologists for Accurate Differential Diagnosis of Skin Diseases in Clinical Images
DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Hupan Lab（湖畔实验室）；Beijing Normal University（北京师范大学）；Sir Run Run Shaw Hospital（邵逸夫医院）；The Second Affiliated Hospital Zhejiang University School of Medicine（浙江大学医学院附属第二医院）；Xiangya Hospital Central South University（中南大学湘雅医院）
Federal University of Espirito Santo, UFES（圣埃斯皮里图联邦大学（UFES））
Skin cancer is one of the most common types of cancer in the world. Different computer-aided diagnosis systems have been proposed to tackle skin lesion diagnosis, most of them based in deep convolutional neural networks. However, recent advances in computer vision achieved state-of-art results in many tasks, notably Transformer-based networks. We explore and evaluate advances in computer vision architectures, training methods and multimodal feature fusion for skin lesion diagnosis task. Experiments show that PiT ($0.800 \pm 0.006$), CoaT ($0.780 \pm 0.024$) and ViT ($0.771 \pm 0.018$) backbone…
Nanyang Technological University（南洋理工大学）；Central South University（中南大学）；Singapore University of Technology and Design（新加坡科技设计大学）
Deep learning techniques have shown their superior performance in dermatologist clinical inspection. Nevertheless, melanoma diagnosis is still a challenging task due to the difficulty of incorporating the useful dermatologist clinical knowledge into the learning process. In this paper, we propose a novel knowledge-aware deep framework that incorporates some clinical knowledge into collaborative learning of two important melanoma diagnosis tasks, i.e., skin lesion segmentation and melanoma recognition. Specifically, to exploit the knowledge of morphological expressions of the lesion region and …
In this paper, we studied extensively on different deep learning based methods to detect melanoma and skin lesion cancers. Melanoma, a form of malignant skin cancer is very threatening to health. Proper diagnosis of melanoma at an earlier stage is crucial for the success rate of complete cure. Dermoscopic images with Benign and malignant forms of skin cancer can be analyzed by computer vision system to streamline the process of skin cancer detection. In this study, we experimented with various neural networks which employ recent deep learning based models like PNASNet-5-Large, InceptionResNetV…
Bangladesh University of Engineering and Technology（孟加拉国工程技术大学）
Melanoma is considered to be the deadliest variant of skin cancer causing around 75\% of total skin cancer deaths. To diagnose Melanoma, clinicians assess and compare multiple skin lesions of the same patient concurrently to gather contextual information regarding the patterns, and abnormality of the skin. So far this concurrent multi-image comparative method has not been explored by existing deep learning-based schemes. In this paper, based on contextual image feature fusion (CIFF), a deep neural network (CIFF-Net) is proposed, which integrates patient-level contextual information into the tr…
University of British Columbia（不列颠哥伦比亚大学）；Simon Fraser University（西蒙弗雷泽大学）
Deep learning models have achieved great success in automating skin lesion diagnosis. However, the ethnic disparity in these models' predictions, where lesions on darker skin types are usually underrepresented and have lower diagnosis accuracy, receives little attention. In this paper, we propose FairDisCo, a disentanglement deep learning framework with contrastive learning that utilizes an additional network branch to remove sensitive attributes, i.e. skin-type information from representations for fairness and another contrastive branch to enhance feature extraction. We compare FairDisCo to t…
Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)
Louisiana State University in Shreveport（路易斯安那州立大学什里夫波特分校）；Kitware（Kitware公司）；Memorial Sloan-Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）；Missouri University of Science and Technology（密苏里科技大学）；Emory University（埃默里大学）；IBM T. J. Watson Research Center（IBM托马斯·J·沃森研究中心）
Pakistan Navy Engineering College (PNEC)（巴基斯坦海军工程学院）；National University of Sciences & Technology (NUST)（国家科技大学）；Electronics and Power Engineering (EPE)（电子与动力工程系）
Technology aided platforms provide reliable tools in almost every field these days. These tools being supported by computational power are significant for applications that need sensitive and precise data analysis. One such important application in the medical field is Automatic Lesion Detection System (ALDS) for skin cancer classification. Computer aided diagnosis helps physicians and dermatologists to obtain a second opinion for proper analysis and treatment of skin cancer. Precise segmentation of the cancerous mole along with surrounding area is essential for proper analysis and diagnosis. …
kin cancer is considered one of the most common type of cancer in several countries. Due to the difficulty and subjectivity in the clinical diagnosis of skin lesions, Computer-Aided Diagnosis systems are being developed for assist experts to perform more reliable diagnosis. The clinical analysis and diagnosis of skin lesions relies not only on the visual information but also on the context information provided by the patient. This work addresses the problem of pigmented skin lesions detection from smartphones captured images. In addition to the features extracted from images, patient context i…
SkinCaRe: A Multimodal Dermatology Dataset Annotated with Medical Caption and Chain-of-Thought Reasoning
School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK–Shenzhen)（数据科学学院，香港中文大学（深圳））；Computer Science Program, CEMSE Division, King Abdullah University of Science and Technology (KAUST)（计算机科学项目，科学与工程学院，国王 Abdullah 科学技术大学）；Center of Excellence on Smart Health, KAUST（智能健康卓越中心，国王 Abdullah 科学技术大学）；Center of Excellence for Generative AI, KAUST（生成式人工智能卓越中心，国王 Abdullah 科学技术大学）；Department of Dermatology, Beijing AnZhen Hospital, Capital Medical University（皮肤科，北京安贞医院，首都医科大学）；Department of Dermatology, Tianjin Institute of Integrative Dermatology, Tianjin Academy of Traditional Chinese Medicine Affiliated Hospital（皮肤科，天津整合皮肤科研究院，天津中医药大学附属医院）；Department of Dermatology, Beijing Aerospace General Hospital（皮肤科，北京航天总医院）；Department of Dermatology, The First Affiliated Hospital, Shantou University Medical College（皮肤科，汕头大学医学院第一附属医院）；DermAssure, LLC（DermAssure 公司）；School of Medicine, New York Medical College（医学院，纽约医学院）；Capital Medical University（首都医科大学）；Department of Dermatology, Second Hospital of Jilin University（皮肤科，吉林大学第二医院）；Emergency Critical Care Center, Beijing AnZhen Hospital, Capital Medical University（急诊重症中心，北京安贞医院，首都医科大学）
Data-driven models are now deployed in a plethora of real-world applications - including automated diagnosis - but models learned from data risk learning biases from that same data. When models learn spurious correlations not found in real-world situations, their deployment for critical tasks, such as medical decisions, can be catastrophic. In this work we address this issue for skin-lesion classification models, with two objectives: finding out what are the spurious correlations exploited by biased networks, and debiasing the models by removing such spurious correlations from them. We perform…
Mpox-AISM: AI-Mediated Super Monitoring for Mpox and Like-Mpox
School of Biomedical Engineering（生物医学工程学院）；Guangzhou Medical University（广州医科大学）；Foshan Sanshui District People’s Hospital（佛山市三水区人民医院）；School of Mathematics and Systems Science（数学与系统科学学院）；Guangdong Polytechnic Normal University（广东技术师范大学）
The segmentation of skin lesions is a crucial task in clinical decision support systems for the computer aided diagnosis of skin lesions. Although deep learning-based approaches have improved segmentation performance, these models are often susceptible to class imbalance in the data, particularly, the fraction of the image occupied by the background healthy skin. Despite variations of the popular Dice loss function being proposed to tackle the class imbalance problem, the Dice loss formulation does not penalize misclassifications of the background pixels. We propose a novel metric-based loss f…
In this study, we investigate what a practically useful approach is in order to achieve robust skin disease diagnosis. A direct approach is to target the ground truth diagnosis labels, while an alternative approach instead focuses on determining skin lesion characteristics that are more visually consistent and discernible. We argue that, for computer-aided skin disease diagnosis, it is both more realistic and more useful that lesion type tags should be considered as the target of an automated diagnosis system such that the system can first achieve a high accuracy in describing skin lesions, an…
SkinGPT-X: A Self-Evolving Collaborative Multi-Agent System for Transparent and Trustworthy Dermatological Diagnosis
School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Department of Dermatology, Tianjin Institute of Integrative Dermatology, Tianjin Academy of Traditional Chinese Medicine Affiliated Hospital（天津中医药大学附属医院天津市中西医结合皮肤病研究所皮肤科）；Department of Dermatology, Beijing AnZhen Hospital, Capital Medical University（首都医科大学附属北京安贞医院皮肤科）
Deutsches Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)（德国人工智能研究中心）；TU Kaiserslautern（凯泽斯劳滕工业大学）
Early detection of skin cancers like melanoma is crucial to ensure high chances of survival for patients. Clinical application of Deep Learning (DL)-based Decision Support Systems (DSS) for skin cancer screening has the potential to improve the quality of patient care. The majority of work in the medical AI community focuses on a diagnosis setting that is mainly relevant for autonomous operation. Practical decision support should, however, go beyond plain diagnosis and provide explanations. This paper provides an overview of works towards explainable, DL-based decision support in medical appli…
Exploring the Challenge and Value of Deep Learning in Automated Skin Disease Diagnosis
Polytechnic Institute, Zhejiang University（浙江大学 polytechnic 院）；School of Social and Public Administration, East China University Of Science And Technology（东华大学社会科学与公共管理学院）；College of Science, National University of Defense Technology（国防科技大学科学学院）；School of Mathematical Sciences, Zhejiang University（浙江大学数学科学学院）
Skin cancer is considered to be the most common human malignancy. Around 5 million new cases of skin cancer are recorded in the United States annually. Early identification and evaluation of skin lesions is of great clinical significance, but the disproportionate dermatologist-patient ratio poses significant problem in most developing nations. Therefore a deep learning based architecture, known as SkiNet, is proposed with an objective to provide faster screening solution and assistance to newly trained physicians in the clinical diagnosis process. The main motive behind Skinet's design and dev…
In Zambia, there is a serious shortage of medical staff where each practitioner attends to about 17000 patients in a given district while still, other patients travel over 10 km to access the basic medical services. In this research, we implement a deep learning model that can perform the clinical diagnosis process. The study will prove whether image analysis is capable of performing clinical diagnosis. It will also enable us to understand if we can use image analysis to lessen the workload on medical practitioners by delegating some tasks to an AI. The success of this study has the potential …
ISTI-CNR（意大利国家研究委员会信息科学与技术研究所）；University of Pisa（比萨大学）；Criteo AI Lab（Criteo人工智能实验室）；Scuola Normale Superiore of Pisa（比萨高等师范学校）
A key issue in critical contexts such as medical diagnosis is the interpretability of the deep learning models adopted in decision-making systems. Research in eXplainable Artificial Intelligence (XAI) is trying to solve this issue. However, often XAI approaches are only tested on generalist classifier and do not represent realistic problems such as those of medical diagnosis. In this paper, we analyze a case study on skin lesion images where we customize an existing XAI approach for explaining a deep learning model able to recognize different types of skin lesions. The explanation is formed by…
AI-Powered Dermatological Diagnosis: From Interpretable Models to Clinical Implementation A Comprehensive Framework for Accessible and Trustworthy Skin Disease Detection
Department of Business Analytics/Data Science Engineering University of New Haven（商业分析/数据科学工程系 罗德岛大学）；Department of Healthcare University of New Haven（医疗健康系 罗德岛大学）
Skin cancer diagnosis using NIR spectroscopy data of skin lesions in vivo using machine learning algorithms
Federal University of Espírito Santo (UFES)（圣埃斯皮里图联邦大学）；Secretary of Health of the Espírito Santo State（圣埃斯皮里图州卫生部）；University Hospital Cassiano Antônio Moraes (HUCAM)（卡西亚诺·安东尼奥·莫赖斯大学医院）
Skin cancer is one of the most threatening diseases worldwide. However, diagnosing skin cancer correctly is challenging. Recently, deep learning algorithms have emerged to achieve excellent performance on various tasks. Particularly, they have been applied to the skin disease diagnosis tasks. In this paper, we present a review on deep learning methods and their applications in skin disease diagnosis. We first present a brief introduction to skin diseases and image acquisition methods in dermatology, and list several publicly available skin datasets for training and testing algorithms. Then, we…
An interpretable imbalanced semi-supervised deep learning framework for improving differential diagnosis of skin diseases
School of Medicine, Xiamen University（厦门大学医学院）；National Institute for Data Science in Health and Medicine, Xiamen University（厦门大学健康医疗大数据国家研究院）；Data Mining Research Center, Xiamen University（厦门大学数据挖掘研究中心）；School of Economics and Management, East China Jiaotong University（华东交通大学经济管理学院）；Department of Pediatrics, the First Affiliated Hospital of Xiamen University（厦门大学附属第一医院儿科）；School of Management, Xiamen University（厦门大学管理学院）；School of Science, Hong Kong University of Science and Technology（香港科技大学理学院）；School of Mathematical Sciences, Ocean University of China（中国海洋大学数学科学学院）；National Economic Engineering Laboratory, Dongbei University of Finance and Economics（东北财经大学国民经济工程实验室）
Dartmouth College（达特茅斯学院）；Dartmouth-Hitchcock Medical Center（达特茅斯-希区柯克医疗中心）；Geisel School of Medicine（盖塞尔医学院）；Metrohealth System（都会医疗系统）；Case Western Reserve University（凯斯西储大学）
Recently, there has been great interest in developing Artificial Intelligence (AI) enabled computer-aided diagnostics solutions for the diagnosis of skin cancer. With the increasing incidence of skin cancers, low awareness among a growing population, and a lack of adequate clinical expertise and services, there is an immediate need for AI systems to assist clinicians in this domain. A large number of skin lesion datasets are available publicly, and researchers have developed AI-based image classification solutions, particularly deep learning algorithms, to distinguish malignant skin lesions fr…
Skin conditions are reported the 4th leading cause of nonfatal disease burden worldwide. However, given the colossal spectrum of skin disorders defined clinically and shortage in dermatology expertise, diagnosing skin conditions in a timely and accurate manner remains a challenging task. Using computer vision technologies, a deep learning system has proven effective assisting clinicians in image diagnostics of radiology, ophthalmology and more. In this paper, we propose a deep learning system (DLS) that may predict differential diagnosis of skin conditions using clinical images. Our DLS formul…
Using Computer Vision for Skin Disease Diagnosis in Bangladesh Enhancing Interpretability and Transparency in Deep Learning Models for Skin Cancer Classification
Hong Kong Baptist University（香港浸会大学）；Harbin Institute of Technology（哈尔滨工业大学）；The University of Hong Kong - Shenzhen Hospital（香港大学深圳医院）；Shenzhen Yiyuan Intelligent Technology Co., Ltd.（深圳怡园智能科技有限公司）
Skin disease is one of the most common types of human diseases, which may happen to everyone regardless of age, gender or race. Due to the high visual diversity, human diagnosis highly relies on personal experience; and there is a serious shortage of experienced dermatologists in many countries. To alleviate this problem, computer-aided diagnosis with state-of-the-art (SOTA) machine learning techniques would be a promising solution. In this paper, we aim at understanding the performance of convolutional neural network (CNN) based approaches. We first build two versions of skin disease datasets…
The utilization of computational photography becomes increasingly essential in the medical field. Today, imaging techniques for dermatology range from two-dimensional (2D) color imagery with a mobile device to professional clinical imaging systems measuring additional detailed three-dimensional (3D) data. The latter are commonly expensive and not accessible to a broad audience. In this work, we propose a novel system and software framework that relies only on low-cost (and even mobile) commodity devices present in every household to measure detailed 3D information of the human skin with a 3D-g…
Coherent Concept-based Explanations in Medical Image and Its Application to Skin Lesion Diagnosis
Universidade da Beira Interior（贝拉内格罗大学）；NOVA LINCS（新里斯本大学计算机科学与信息学实验室）；INESC TEC（葡萄牙科技与工程计算系统研究所）；Faculdade de Engenharia da Universidade do Porto（波尔图大学工程学院）
Skin Lesion Diagnosis using Ensembles, Unscaled Multi-Crop Evaluation and Loss Weighting
Forschungszentrum Medizintechnik Hamburg（汉堡医疗技术研究中心）；Hamburg University of Technology（汉堡工业大学）；University Medical Center Hamburg-Eppendorf（汉堡-埃彭多夫大学医学中心）
The identification of melanoma involves an integrated analysis of skin lesion images acquired using the clinical and dermoscopy modalities. Dermoscopic images provide a detailed view of the subsurface visual structures that supplement the macroscopic clinical images. Melanoma diagnosis is commonly based on the 7-point visual category checklist (7PC). The 7PC contains intrinsic relationships between categories that can aid classification, such as shared features, correlations, and the contributions of categories towards diagnosis. Manual classification is subjective and prone to intra- and inte…
Concept-Attention Whitening for Interpretable Skin Lesion Diagnosis
The Hong Kong University of Science and Technology（香港科技大学）；Fudan University（复旦大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深圳研究院）
