Transrectal biopsies under 2D ultrasound (US) control are the current clinical standard for prostate cancer diagnosis. The isoechogenic nature of prostate carcinoma makes it necessary to sample the gland systematically, resulting in a low sensitivity. Also, it is difficult for the clinician to follow the sampling protocol accurately under 2D US control and the exact anatomical location of the biopsy cores is unknown after the intervention. Tracking systems for prostate biopsies make it possible to generate biopsy distribution maps for intra- and post-interventional quality control and 3D visualisation of histological results for diagnosis and treatment planning. They can also guide the clinician toward non-ultrasound targets. In this paper, a volume-swept 3D US based tracking system for fast and accurate estimation of prostate tissue motion is proposed. The entirely image-based system solves the patient motion problem with an a priori model of rectal probe kinematics. Prostate deformations are estimated with elastic registration to maximize accuracy. The system is robust with only 17 registration failures out of 786 (2%) biopsy volumes acquired from 47 patients during biopsy sessions. Accuracy was evaluated to 0.76$\pm$0.52mm using manually segmented fiducials on 687 registered volumes stemming from 40 patients. A clinical protocol for assisted biopsy acquisition was designed and implemented as a biopsy assistance system, which allows to overcome the draw-backs of the standard biopsy procedure.
Massachusetts Institute of Technology（麻省理工学院）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）
Histopathology tissue samples are widely available in two states: paraffin-embedded unstained and non-paraffin-embedded stained whole slide RGB images (WSRI). Hematoxylin and eosin stain (H&E) is one of the principal stains in histology but suffers from several shortcomings related to tissue preparation, staining protocols, slowness and human error. We report two novel approaches for training machine learning models for the computational H&E staining and destaining of prostate core biopsy RGB images. The staining model uses a conditional generative adversarial network that learns hierarchical …
Deep Learning for Generating Computational PIN-4 Immunohistochemistry Staining from Prostate Biopsy H&E Images
Biomedical Engineering University of California, Irvine Irvine, CA, USA（生物医学工程 卡罗来纳大学伊文城分校 伊文城，加州，美国）；Laboratory Medicine Biomedical Engineering Electrical Engineering（实验室医学 生物医学工程 电气工程）；Computer Science University of California, Irvine Irvine, CA, USA（计算机科学 卡罗来纳大学伊文城分校 伊文城，加州，美国）
Clinically significant prostate cancer has a better chance to be sampled during ultrasound-guided biopsy procedures, if suspected lesions found in pre-operative magnetic resonance (MR) images are used as targets. However, the diagnostic accuracy of the biopsy procedure is limited by the operator-dependent skills and experience in sampling the targets, a sequential decision making process that involves navigating an ultrasound probe and placing a series of sampling needles for potentially multiple targets. This work aims to learn a reinforcement learning (RL) policy that optimises the actions o…
Background: Transrectal ultrasound guided systematic biopsies of the prostate is a routine procedure to establish a prostate cancer diagnosis. However, the 10-12 prostate core biopsies only sample a relatively small volume of the prostate, and tumour lesions in regions between biopsy cores can be missed, leading to a well-known low sensitivity to detect clinically relevant cancer. As a proof-of-principle, we developed and validated a deep convolutional neural network model to distinguish between morphological patterns in benign prostate biopsy whole slide images from men with and without estab…
Queen’s University（女王大学）；University of British Columbia（不列颠哥伦比亚大学）；Exact Imaging
MOTIVATION: Detection of prostate cancer during transrectal ultrasound-guided biopsy is challenging. The highly heterogeneous appearance of cancer, presence of ultrasound artefacts, and noise all contribute to these difficulties. Recent advancements in high-frequency ultrasound imaging - micro-ultrasound - have drastically increased the capability of tissue imaging at high resolution. Our aim is to investigate the development of a robust deep learning model specifically for micro-ultrasound-guided prostate cancer biopsy. For the model to be clinically adopted, a key challenge is to design a so…
Queen’s University（女王大学）；University of British Columbia（不列颠哥伦比亚大学）；Exact Imaging（Exact Imaging公司）
A large body of previous machine learning methods for ultrasound-based prostate cancer detection classify small regions of interest (ROIs) of ultrasound signals that lie within a larger needle trace corresponding to a prostate tissue biopsy (called biopsy core). These ROI-scale models suffer from weak labeling as histopathology results available for biopsy cores only approximate the distribution of cancer in the ROIs. ROI-scale models do not take advantage of contextual information that are normally considered by pathologists, i.e. they do not consider information about surrounding tissue and …
Virginia Polytechnic Institute and State University（弗吉尼亚理工大学）；Xi’an University of Architecture and Technology（西安建筑科技大学）
A statistical volumetric model, showing the probability map of localized prostate cancer within the host anatomical structure, has been developed from 90 optically-imaged surgical specimens. This master model permits an accurate characterization of prostate cancer distribution patterns and an atlas-informed biopsy sampling strategy. The model is constructed by mapping individual prostate models onto a site model, together with localized tumors. An accurate multi-object non-rigid warping scheme is developed based on a mixture of principal-axis registrations. We report our evaluation and pilot s…
The PAR dataset: Prostate biopsy whole slide images from an underrepresented Middle Eastern population
Koya University（科亚大学）；Karolinska Institutet（卡罗林斯卡学院）；Hawler Medical University（哈勒医科大学）；PAR Private Hospital（PAR私立医院）；University of Sulaimani（苏莱曼尼亚大学）；Stavanger University Hospital（斯塔万格大学医院）；University of Stavanger（斯塔万格大学）；ELLIS Institute Finland（芬兰ELLIS研究所）；University of Turku（图尔库大学）
Karolinska Institutet（卡罗林斯卡学院）；Tampere University（坦佩雷大学）；University of Turku（图尔库大学）；S:t Göran Hospital（圣戈兰医院）；Aquesta Uropathology（阿奎斯塔泌尿病理中心）；University of Queensland（昆士兰大学）；University of Otago（奥塔哥大学）；Wellington School of Medicine and Health Sciences（惠灵顿医学与健康科学学院）；Aichi Medical University（爱知医科大学）
Background: The detection of perineural invasion (PNI) by carcinoma in prostate biopsies has been shown to be associated with poor prognosis. The assessment and quantification of PNI is; however, labor intensive. In the study we aimed to develop an algorithm based on deep neural networks to aid pathologists in this task. Methods: We collected, digitized and pixel-wise annotated the PNI findings in each of the approximately 80,000 biopsy cores from the 7,406 men who underwent biopsy in the prospective and diagnostic STHLM3 trial between 2012 and 2014. In total, 485 biopsy cores showed PNI. We a…
Massachusetts Institute of Technology（麻省理工学院）；Harvard Medical School（哈佛医学院）；Boston University School of Medicine（波士顿大学医学院）；VA Boston Healthcare（波士顿退伍军人医疗中心）
Histopathological diagnoses of tumors in tissue biopsy after Hematoxylin and Eosin (H&E) staining is the gold standard for oncology care. H&E staining is slow and uses dyes, reagents and precious tissue samples that cannot be reused. Thousands of native nonstained RGB Whole Slide Image (RWSI) patches of prostate core tissue biopsies were registered with their H&E stained versions. Conditional Generative Adversarial Neural Networks (cGANs) that automate conversion of native nonstained RWSI to computational H&E stained images were then trained. High similarities between computational and H&E dye…
Northumbria University at Newcastle（纽卡斯尔诺森比亚大学）；College of Computer and Information Sciences, King Saud University（沙特国王大学计算机与信息科学学院）
Colorectal and prostate cancers are the most common types of cancer in men worldwide. To diagnose colorectal and prostate cancer, a pathologist performs a histological analysis on needle biopsy samples. This manual process is time-consuming and error-prone, resulting in high intra and interobserver variability, which affects diagnosis reliability. This study aims to develop an automatic computerized system for diagnosing colorectal and prostate tumors by using images of biopsy samples to reduce time and diagnosis error rates associated with human analysis. We propose a CNN model for classifyin…
An end-to-end-trained vision-language model for native-language prostate pathology report generation
Technical University of Munich（慕尼黑工业大学）；Munich Data Science Institute（慕尼黑数据科学研究所）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University Hospital Heidelberg（海德堡大学医院）；Heidelberg University（海德堡大学）；Interdisciplinary Center for Scientific Computing (IWR)（跨学科科学计算中心（IWR））
Queen’s University（女王大学）；University of British Columbia（不列颠哥伦比亚大学）；Exact Imaging（Exact Imaging公司）
Deep learning-based analysis of high-frequency, high-resolution micro-ultrasound data shows great promise for prostate cancer detection. Previous approaches to analysis of ultrasound data largely follow a supervised learning paradigm. Ground truth labels for ultrasound images used for training deep networks often include coarse annotations generated from the histopathological analysis of tissue samples obtained via biopsy. This creates inherent limitations on the availability and quality of labeled data, posing major challenges to the success of supervised learning methods. On the other hand, …
The emergence of multi-parametric magnetic resonance imaging (mpMRI) has had a profound impact on the diagnosis of prostate cancers (PCa), which is the most prevalent malignancy in males in the western world, enabling a better selection of patients for confirmation biopsy. However, analyzing these images is complex even for experts, hence opening an opportunity for computer-aided diagnosis systems to seize. This paper proposes a fully automatic system based on Deep Learning that takes a prostate mpMRI from a PCa-suspect patient and, by leveraging the Retina U-Net detection framework, locates P…
Towards Multi-modality Fusion and Prototype-based Feature Refinement for Clinically Significant Prostate Cancer Classification in Transrectal Ultrasound
School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University（深圳大学医学部生物医学工程学院）；Sun Yat-sen University Cancer Center（中山大学肿瘤防治中心）
Mask Enhanced Deeply Supervised Prostate Cancer Detection on B-mode Micro-Ultrasound
Stanford University（斯坦福大学）；The Catholic University of Korea（韩国天主教大学）；University of Alberta（阿尔伯塔大学）；University of California Los Angeles（加州大学洛杉矶分校）；Humanitas University（乌尼塔斯大学）；IRCCS Humanitas Research Hospital（IRCCS乌尼塔斯研究医院）；University of Florida（佛罗里达大学）；University of California San Diego（加州大学圣地亚哥分校）
MrTrack: Register Mamba for Needle Tracking with Rapid Reciprocating Motion during Ultrasound-Guided Aspiration Biopsy
Department of Mechanical and Automation Engineering（机械与自动化工程系）；Department of Computer Science and Engineering（计算机科学与工程系）；Department of Medicine and Therapeutics and Institute of Digestive Disease（医学与治疗学系和消化疾病研究所）；The Chinese University of Hong Kong（香港中文大学）
Hybrid-View Attention Network for Clinically Significant Prostate Cancer Classification in Transrectal Ultrasound
Smart Medical Imaging, Learning and Engineering (SMILE) Lab（智能医学影像、学习与工程实验室）；Medical UltraSound Image Computing (MUSIC) Lab（超声影像计算（MUSIC）实验室）；School of Biomedical Engineering（生物医学工程学院）；Shenzhen University Medical School（深圳大学医学院）；Shenzhen University（深圳大学）；The Department of Ultrasound（超声科）；State Key Laboratory of Oncology in South China（南方肿瘤学关键实验室）；Guangdong Provincial Clinical Research Center for Cancer（广东省癌症临床研究中心）；Sun Yat-sen University Cancer Center（中山大学肿瘤中心）
Weakly Supervised Spatial Grounding for Discriminative Attention-Based Ultrasound-Histopathology Alignment in Prostate Cancer Grading
University of British Columbia（不列颠哥伦比亚大学）；Vector Institute（向量研究所）；Ordensklinikum Linz（林茨教团医院）；University of Alberta（阿尔伯塔大学）；Exact Imaging（Exact Imaging公司）
The size and geometry of the prostate are known to be pivotal quantities used by clinicians to assess the condition of the gland during prostate cancer screening. As an alternative to palpation, an increasing number of methods for estimation of the above-mentioned quantities are based on using imagery data of prostate. The necessity to process large volumes of such data creates a need for automatic segmentation tools which would allow the estimation to be carried out with maximum accuracy and efficiency. In particular, the use of transrectal ultrasound (TRUS) imaging in prostate cancer screeni…
Karolinska Institutet（卡罗林斯卡医学院）；Tampere University（坦佩雷大学）；Uppsala University（乌普萨拉大学）；University of Otago（奥塔哥大学）；Queen Mary University of London（伦敦玛丽女王大学）；Bostwick Laboratories（博斯特威克实验室）；University Health Network（大学健康网络）；Indiana University School of Medicine（印第安纳大学医学院）；Yale University School of Medicine（耶鲁大学医学院）；Medical College of Wisconsin（威斯康星医学院）；Royal Prince Alfred Hospital（阿尔弗雷德王子医院）；University of Sydney（悉尼大学）；University Hospital Bonn（波恩大学医院）；University of São Paulo Medical School（圣保罗大学医学院）；Cleveland Clinic（克利夫兰医学中心）；Southmead Hospital（索斯米德医院）；Taipei Veterans General Hospital（台北荣民总医院）；Aquesta Uropathology（Aquesta泌尿病理中心）；University of Queensland（昆士兰大学）；University of Toronto（多伦多大学）；Jikei University School of Medicine（慈惠会医科大学）；Aichi Medical University（爱知医科大学）；University Hospital of Wales（威尔士大学医院）；UT Southwestern Medical Center（德克萨斯大学西南医学中心）；SciLifeLab（生命科学实验室）；S:t Göran Hospital（圣约兰医院）
Background: An increasing volume of prostate biopsies and a world-wide shortage of uro-pathologists puts a strain on pathology departments. Additionally, the high intra- and inter-observer variability in grading can result in over- and undertreatment of prostate cancer. Artificial intelligence (AI) methods may alleviate these problems by assisting pathologists to reduce workload and harmonize grading. Methods: We digitized 6,682 needle biopsies from 976 participants in the population based STHLM3 diagnostic study to train deep neural networks for assessing prostate biopsies. The networks were …
Siemens Healthineers（西门子医疗）；Imperial College London（帝国理工学院）
Real-time localization of prostate gland in trans-rectal ultrasound images is a key technology that is required to automate the ultrasound guided prostate biopsy procedures. In this paper, we propose a new deep learning based approach which is aimed at localizing several prostate landmarks efficiently and robustly. We propose a multitask learning approach primarily to make the overall algorithm more contextually aware. In this approach, we not only consider the explicit learning of landmark locations, but also build-in a mechanism to learn the contour of the prostate. This multitask learning i…
Learning Prostate Anatomy at Test Time for Cancer Detection in Micro-Ultrasound
University of British Columbia（不列颠哥伦比亚大学）；Vector Institute（向量研究所）；Ordensklinikum Linz（林茨 Ordenklinikum 医院）；University of Alberta（阿尔伯塔大学）；Exact Imaging（Exact Imaging 公司）
Stanford University（斯坦福大学）；University College London（伦敦大学学院）；University of California Los Angeles（加利福尼亚大学洛杉矶分校）
Prostate biopsy and image-guided treatment procedures are often performed under the guidance of ultrasound fused with magnetic resonance images (MRI). Accurate image fusion relies on accurate segmentation of the prostate on ultrasound images. Yet, the reduced signal-to-noise ratio and artifacts (e.g., speckle and shadowing) in ultrasound images limit the performance of automated prostate segmentation techniques and generalizing these methods to new image domains is inherently difficult. In this study, we address these challenges by introducing a novel 2.5D deep neural network for prostate segm…
ContextVision AB（ContextVision公司）；Karolinska Institutet and University Hospital（卡罗林斯卡学院暨大学医院）
Gleason grading specified in ISUP 2014 is the clinical standard in staging prostate cancer and the most important part of the treatment decision. However, the grading is subjective and suffers from high intra and inter-user variability. To improve the consistency and objectivity in the grading, we introduced glandular tissue WithOut Basal cells (WOB) as the ground truth. The presence of basal cells is the most accepted biomarker for benign glandular tissue and the absence of basal cells is a strong indicator of acinar prostatic adenocarcinoma, the most common form of prostate cancer. Glandular…
FRAMOS GmbH（FRAMOS有限公司）；Johns Hopkins University（约翰斯·霍普金斯大学）；Technische Universität München（慕尼黑工业大学）
Tracking of rotation and translation of medical instruments plays a substantial role in many modern interventions. Traditional external optical tracking systems are often subject to line-of-sight issues, in particular when the region of interest is difficult to access or the procedure allows only for limited rigid body markers. The introduction of inside-out tracking systems aims to overcome these issues. We propose a marker-less tracking system based on visual SLAM to enable tracking of instruments in an interventional scenario. To achieve this goal, we mount a miniature multi-modal (monocula…
Rensselaer Polytechnic Institute（伦斯勒理工学院）；National Institutes of Health（美国国立卫生研究院）
Fusing intra-operative 2D transrectal ultrasound (TRUS) image with pre-operative 3D magnetic resonance (MR) volume to guide prostate biopsy can significantly increase the yield. However, such a multimodal 2D/3D registration problem is a very challenging task. In this paper, we propose an end-to-end frame-to-volume registration network (FVR-Net), which can efficiently bridge the previous research gaps by aligning a 2D TRUS frame with a 3D TRUS volume without requiring hardware tracking. The proposed FVR-Net utilizes a dual-branch feature extraction module to extract the information from TRUS fr…
Rensselaer Polytechnic Institute（伦斯勒理工学院）；National Institutes of Health（美国国立卫生研究院）
Reconstructed 3D ultrasound volume provides more context information compared to a sequence of 2D scanning frames, which is desirable for various clinical applications such as ultrasound-guided prostate biopsy. Nevertheless, 3D volume reconstruction from freehand 2D scans is a very challenging problem, especially without the use of external tracking devices. Recent deep learning based methods demonstrate the potential of directly estimating inter-frame motion between consecutive ultrasound frames. However, such algorithms are specific to particular transducers and scanning trajectories associa…
Rensselaer Polytechnic Institute（伦斯勒理工学院）；National Institutes of Health（美国国立卫生研究院）；National Cancer Institute（美国国家癌症研究所）
Prostate cancer biopsy benefits from accurate fusion of transrectal ultrasound (TRUS) and magnetic resonance (MR) images. In the past few years, convolutional neural networks (CNNs) have been proved powerful in extracting image features crucial for image registration. However, challenging applications and recent advances in computer vision suggest that CNNs are quite limited in its ability to understand spatial correspondence between features, a task in which the self-attention mechanism excels. This paper aims to develop a self-attention mechanism specifically for cross-modal image registrati…
Hamburg University of Technology（汉堡工业大学）；University Hospital Schleswig-Holstein（石勒苏益格-荷尔斯泰因大学医院）
Needle insertion is common during minimally invasive interventions such as biopsy or brachytherapy. During soft tissue needle insertion, forces acting at the needle tip cause tissue deformation and needle deflection. Accurate needle tip force measurement provides information on needle-tissue interaction and helps detecting and compensating potential misplacement. For this purpose we introduce an image-based needle tip force estimation method using an optical fiber imaging the deformation of an epoxy layer below the needle tip over time. For calibration and force estimation, we introduce a nove…
Rensselaer Polytechnic Institute（伦斯勒理工学院）；National Institutes of Health（美国国立卫生研究院）
Transrectal ultrasound (US) is the most commonly used imaging modality to guide prostate biopsy and its 3D volume provides even richer context information. Current methods for 3D volume reconstruction from freehand US scans require external tracking devices to provide spatial position for every frame. In this paper, we propose a deep contextual learning network (DCL-Net), which can efficiently exploit the image feature relationship between US frames and reconstruct 3D US volumes without any tracking device. The proposed DCL-Net utilizes 3D convolutions over a US video segment for feature extra…
Poisson Ordinal Network for Gleason Group Estimation Using Bi-Parametric MRI
Centre for Medical Image Computing, University College London（伦敦大学学院医学影像计算中心）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Centre for Medical Imaging, University College London（伦敦大学学院医学影像中心）；National Institute of Health Data Science, Peking University（北京大学健康数据科学国家研究院）
Hamburg University of Technology（汉堡工业大学）；University Hospital Schleswig-Holstein（石勒苏益格-荷尔斯泰因大学医院）
Purpose. Precise placement of needles is a challenge in a number of clinical applications such as brachytherapy or biopsy. Forces acting at the needle cause tissue deformation and needle deflection which in turn may lead to misplacement or injury. Hence, a number of approaches to estimate the forces at the needle have been proposed. Yet, integrating sensors into the needle tip is challenging and a careful calibration is required to obtain good force estimates. Methods. We describe a fiber-optical needle tip force sensor design using a single OCT fiber for measurement. The fiber images the defo…
