Each woman living in the United States has about 1 in 8 chance of developing invasive breast cancer. The mitotic cell count is one of the most common tests to assess the aggressiveness or grade of breast cancer. In this prognosis, histopathology images must be examined by a pathologist using high-resolution microscopes to count the cells. Unfortunately, this can be an exhaustive task with poor reproducibility, especially for non-experts. Deep learning networks have recently been adapted to medical applications which are able to automatically localize these regions of interest. However, these region-based networks lack the ability to take advantage of the segmentation features produced by a full image CNN which are often used as a sole method of detection. Therefore, the proposed method leverages Faster RCNN for object detection while fusing segmentation features generated by a UNet with RGB image features to achieve an F-score of 0.508 on the MITOS-ATYPIA 2014 mitosis counting challenge dataset, outperforming state-of-the-art methods.
We present a unified framework to predict tumor proliferation scores from breast histopathology whole slide images. Our system offers a fully automated solution to predicting both a molecular data-based, and a mitosis counting-based tumor proliferation score. The framework integrates three modules, each fine-tuned to maximize the overall performance: An image processing component for handling whole slide images, a deep learning based mitosis detection network, and a proliferation scores prediction module. We have achieved 0.567 quadratic weighted Cohen's kappa in mitosis counting-based score p…
IBM Research Zurich（IBM苏黎世研究院）；ETH Zurich（苏黎世联邦理工学院）
Mitotic counting is a vital prognostic marker of tumor proliferation in breast cancer. Deep learning-based mitotic detection is on par with pathologists, but it requires large labeled data for training. We propose a deep classification framework for enhancing mitosis detection by leveraging class label information, via softmax loss, and spatial distribution information among samples, via distance metric learning. We also investigate strategies towards steadily providing informative samples to boost the learning. The efficacy of the proposed framework is established through evaluation on ICPR 2…
University of Warwick（华威大学）；University of Oulu（奥卢大学）；The Alan Turing Institute（艾伦·图灵研究所）；Aalto University（阿尔托大学）
Mitosis count is an important biomarker for prognosis of various cancers. At present, pathologists typically perform manual counting on a few selected regions of interest in breast whole-slide-images (WSIs) of patient biopsies. This task is very time-consuming, tedious and subjective. Automated mitosis detection methods have made great advances in recent years. However, these methods require exhaustive labeling of a large number of selected regions of interest. This task is very expensive because expert pathologists are needed for reliable and accurate annotations. In this paper, we present a …
University Joseph Fourier（约瑟夫·傅里叶大学）；IPAL, CNRS UMI 2955（IPAL，法国国家科学研究中心UMI 2955）；CoSMo Software（CoSMo软件公司）；Temasys Communications（Temasys通信公司）；University Pierre and Marie Curie（皮埃尔和玛丽·居里大学）
Accurate detection of mitosis plays a critical role in breast cancer histopathology. Manual detection and counting of mitosis is tedious and subject to considerable inter- and intra-reader variations. Multispectral imaging is a recent medical imaging technology, proven successful in increasing the segmentation accuracy in other fields. This study aims at improving the accuracy of mitosis detection by developing a specific solution using multispectral and multifocal imaging of breast cancer histopathological data. We propose to enable clinical routine-compliant quality of mitosis discrimination…
Universitat Politècnica de València（瓦伦西亚理工大学）；Stavanger University Hospital（斯塔万格大学医院）；University of Stavanger（斯塔万格大学）
Mitotic activity is a crucial proliferation biomarker for the diagnosis and prognosis of different types of cancers. Nevertheless, mitosis counting is a cumbersome process for pathologists, prone to low reproducibility, due to the large size of augmented biopsy slides, the low density of mitotic cells, and pattern heterogeneity. To improve reproducibility, deep learning methods have been proposed in the last years using convolutional neural networks. However, these methods have been hindered by the process of data labelling, which usually solely consist of the mitosis centroids. Therefore, cur…
Radboud University Medical Center（拉德堡德大学医学中心）；Canisius-Wilhelmina Hospital（卡尼修斯-威廉明娜医院）；Jeroen Bosch Hospital（耶伦博斯医院）
Manual counting of mitotic tumor cells in tissue sections constitutes one of the strongest prognostic markers for breast cancer. This procedure, however, is time-consuming and error-prone. We developed a method to automatically detect mitotic figures in breast cancer tissue sections based on convolutional neural networks (CNNs). Application of CNNs to hematoxylin and eosin (H&E) stained histological tissue sections is hampered by: (1) noisy and expensive reference standards established by pathologists, (2) lack of generalization due to staining variation across laboratories, and (3) high compu…
Image Sciences Institute, University Medical Center Utrecht（乌得勒支大学医学中心影像科学研究所）；Pathology Department, University Medical Center Utrecht（乌得勒支大学医学中心病理系）；BME Department, Case Western Reserve University（凯斯西储大学生物医学工程系）；MindLab, National University of Colombia（哥伦比亚国立大学思维实验室）；Department of Applied Mathematics and Computer Science, Technical University of Denmark（丹麦技术大学应用数学与计算机科学系）；IDSIA, USI-SUPSI（瑞士应用科技大学智能系统与数据分析研究所）；Department of Computer Engineering, Işık University（伊希克大学计算机工程系）；MINES ParisTech, PSL-Research University, CBIO-Centre for Computational Biology（巴黎高等矿业学院PSL研究大学生物计算中心）；Institut Curie（居里研究所）；INSERM U900（法国国家健康与医学研究院U900）；Graduate Institute of Biomedical Engineering, National Taiwan University of Science and Technology（台湾科技大学生物医学工程研究所）；Corporate R&D Center, Panasonic Healthcare Co., Ltd.（松下医疗保健公司企业研发中心）；School of Computing, Engineering and Physical Sciences, University of Central Lancashire（中央兰开夏大学计算、工程与物理科学学院）；I3S, UMR 7271 UNS-CNRS, Nice Sophia-Antipolis University（尼斯索菲亚安蒂波利斯大学I3S联合研究实验室）；CVSSP, University of Surrey（萨里大学视觉、声音和感知安全研究中心）；Department of Computer Science and Sheffield institute for Translational Medicine, University of Sheffield（谢菲尔德大学计算机科学与谢菲尔德转化医学研究所）；Department of Computer Science, University of Warwick（华威大学计算机系）；Department of Computer Science & Engineering, College of Engineering Qatar University（卡塔尔大学工程学院计算机科学与工程系）；The Michael Letcher Department of Cellular Pathology, Princess Alexandra Hospital NHS Trust（公主亚历山医院NHS信托迈克尔·莱切尔细胞病理学系）
The proliferative activity of breast tumors, which is routinely estimated by counting of mitotic figures in hematoxylin and eosin stained histology sections, is considered to be one of the most important prognostic markers. However, mitosis counting is laborious, subjective and may suffer from low inter-observer agreement. With the wider acceptance of whole slide images in pathology labs, automatic image analysis has been proposed as a potential solution for these issues. In this paper, the results from the Assessment of Mitosis Detection Algorithms 2013 (AMIDA13) challenge are described. The …
Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）；School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）
Recently, RGB-Thermal based perception has shown significant advances. Thermal information provides useful clues when visual cameras suffer from poor lighting conditions, such as low light and fog. However, how to effectively fuse RGB images and thermal data remains an open challenge. Previous works involve naive fusion strategies such as merging them at the input, concatenating multi-modality features inside models, or applying attention to each data modality. These fusion strategies are straightforward yet insufficient. In this paper, we propose a novel fusion method named Explicit Attention…
Crowd counting in single-view images has achieved outstanding performance on existing counting datasets. However, single-view counting is not applicable to large and wide scenes (e.g., public parks, long subway platforms, or event spaces) because a single camera cannot capture the whole scene in adequate detail for counting, e.g., when the scene is too large to fit into the field-of-view of the camera, too long so that the resolution is too low on faraway crowds, or when there are too many large objects that occlude large portions of the crowd. Therefore, to solve the wide-area counting task r…
A Transformer-based Multimodal Fusion Model for Efficient Crowd Counting Using Visual and Wireless Signals
School of Electrical and Electronic Engineering, University College Dublin（都柏林大学电子与电气工程学院）；College of Electrical Engineering and Automation, Shandong University of Science and Technology（山东科技大学电气工程与自动化学院）
FusionCounting: Robust visible-infrared image fusion guided by crowd counting via multi-task learning
School of Information Science and Engineering, Yanshan University（燕山大学信息科学与工程学院）；Fusion Intelligence Laboratory, Department of Computer Science, University of Exeter（埃克塞特大学计算机科学系融合智能实验室）
University of Kansas（堪萨斯大学）；Ryerson University（瑞尔森大学）
Crowd counting in still images is a challenging problem in practice due to huge crowd-density variations, large perspective changes, severe occlusion, and variable lighting conditions. The state-of-the-art patch rescaling module (PRM) based approaches prove to be very effective in improving the crowd counting performance. However, the PRM module requires an additional and compromising crowd-density classification process. To address these issues and challenges, the paper proposes a new multi-resolution fusion based end-to-end crowd counting network. It employs three deep-layers based columns/b…
University of Warwick（华威大学）；University of Glasgow（格拉斯哥大学）
State-of-the-art crowd counting models follow an encoder-decoder approach. Images are first processed by the encoder to extract features. Then, to account for perspective distortion, the highest-level feature map is fed to extra components to extract multiscale features, which are the input to the decoder to generate crowd densities. However, in these methods, features extracted at earlier stages during encoding are underutilised, and the multiscale modules can only capture a limited range of receptive fields, albeit with considerable computational cost. This paper proposes a novel crowd count…
RGB-Thermal (RGB-T) crowd counting is a challenging task, which uses thermal images as complementary information to RGB images to deal with the decreased performance of unimodal RGB-based methods in scenes with low-illumination or similar backgrounds. Most existing methods propose well-designed structures for cross-modal fusion in RGB-T crowd counting. However, these methods have difficulty in encoding cross-modal contextual semantic information in RGB-T image pairs. Considering the aforementioned problem, we propose a two-stream RGB-T crowd counting network called Multi-Attention Fusion Netwo…
Johns Hopkins University（约翰斯·霍普金斯大学）；Department of Electrical and Computer Engineering（电气与计算机工程系）
Crowd counting presents enormous challenges in the form of large variation in scales within images and across the dataset. These issues are further exacerbated in highly congested scenes. Approaches based on straightforward fusion of multi-scale features from a deep network seem to be obvious solutions to this problem. However, these fusion approaches do not yield significant improvements in the case of crowd counting in congested scenes. This is usually due to their limited abilities in effectively combining the multi-scale features for problems like crowd counting. To overcome this, we focus…
Image Analysis in Medicine Lab (IAMLAB), Electrical, Computer and Biomedical Engineering, Ryerson University（Ryerson大学电气、计算机与生物医学工程学院医学影像分析实验室（IAMLAB））；Princess Margaret Cancer Centre, University Health Network（玛格丽特公主癌症中心（大学健康网络））；Department of Laboratory Medicine and Pathobiology, University of Toronto（多伦多大学检验医学与病理生物学系）
Mitotic counts are one of the key indicators of breast cancer prognosis. However, accurate mitotic cell counting is still a difficult problem and is labourious. Automated methods have been proposed for this task, but are usually dependent on the training images and show poor performance on unseen domains. In this work, we present a multi-stage mitosis detection method based on a Cascade RCNN developed to be sequentially more selective against false positives. On the preliminary test set, the algorithm scores an F1-score of 0.7492.
In recent years, crowd counting, a technique for predicting the number of people in an image, becomes a challenging task in computer vision. In this paper, we propose a cross-column feature fusion network to solve the problem of information redundancy in columns. We introduce the Information Fusion Module (IFM) which provides a channel for information flow to help different columns to obtain significant information from another column. Through this channel, different columns exchange information with each other and extract useful features from the other column to enhance key information. Hence…
Semi-Supervised Multi-View Crowd Counting by Ranking Multi-View Fusion Models
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）
The task of crowd counting in varying density scenes is an extremely difficult challenge due to large scale variations. In this paper, we propose a novel dual path multi-scale fusion network architecture with attention mechanism named SFANet that can perform accurate count estimation as well as present high-resolution density maps for highly congested crowd scenes. The proposed SFANet contains two main components: a VGG backbone convolutional neural network (CNN) as the front-end feature map extractor and a dual path multi-scale fusion networks as the back-end to generate density map. These du…
Deep Learning for Semen Analysis in Male Infertility: Computer Vision, Multimodal Fusion, and Clinical Translation
Department of Gynaecology and Obstetrics, The Affiliated Jiangyin Hospital of Nantong University（南通大学附属江阴医院妇产科）；Department of Oncology, the Affiliated Jiangyin Hospital of Nantong University（南通大学附属江阴医院肿瘤科）；Thrust of AI, Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）人工智能推力实验室）；FertiTech AI（生殖科技人工智能公司）；Department of Oncology, Suzhou Xiangcheng People’s Hospital（苏州市相城区人民医院肿瘤科）；Department of Biological Sciences and Bioinformatics, School of Science, Xi’an Jiaotong-Liverpool University（西交利物浦大学理学院生物科学与生物信息学系）；School of Artificial Intelligence and Computer Science, Nantong University（南通大学人工智能与计算机科学学院）
In this paper, we propose a three-stream adaptive fusion network named TAFNet, which uses paired RGB and thermal images for crowd counting. Specifically, TAFNet is divided into one main stream and two auxiliary streams. We combine a pair of RGB and thermal images to constitute the input of main stream. Two auxiliary streams respectively exploit RGB image and thermal image to extract modality-specific features. Besides, we propose an Information Improvement Module (IIM) to fuse the modality-specific features into the main stream adaptively. Experiment results on RGBT-CC dataset show that our me…
National Yang Ming Chiao Tung University（国立阳明交通大学）；University at Albany, State University of New York（纽约州立大学奥尔巴尼分校）
We develop a Synthetic Fusion Pyramid Network (SPF-Net) with a scale-aware loss function design for accurate crowd counting. Existing crowd-counting methods assume that the training annotation points were accurate and thus ignore the fact that noisy annotations can lead to large model-learning bias and counting error, especially for counting highly dense crowds that appear far away. To the best of our knowledge, this work is the first to properly handle such noise at multiple scales in end-to-end loss design and thus push the crowd counting state-of-the-art. We model the noise of crowd annotat…
University of Kansas（堪萨斯大学）；Ryerson University（瑞尔森大学）
The paper focuses on improving the recent plug-and-play patch rescaling module (PRM) based approaches for crowd counting. In order to make full use of the PRM potential and obtain more reliable and accurate results for challenging images with crowd-variation, large perspective, extreme occlusions, and cluttered background regions, we propose a new PRM based multi-resolution and multi-task crowd counting network by exploiting the PRM module with more effectiveness and potency. The proposed model consists of three deep-layered branches with each branch generating feature maps of different resolu…
Crowd counting aims to estimate the number of persons in a scene. Most state-of-the-art crowd counting methods based on color images can't work well in poor illumination conditions due to invisible objects. With the widespread use of infrared cameras, crowd counting based on color and thermal images is studied. Existing methods only achieve multi-modal fusion without count objective constraint. To better excavate multi-modal information, we use count-guided multi-modal fusion and modal-guided count enhancement to achieve the impressive performance. The proposed count-guided multi-modal fusion …
Reliable seed yield estimation is an indispensable step in plant breeding programs geared towards cultivar development in major row crops. The objective of this study is to develop a machine learning (ML) approach adept at soybean [\textit{Glycine max} L. (Merr.)] pod counting to enable genotype seed yield rank prediction from in-field video data collected by a ground robot. To meet this goal, we developed a multi-view image-based yield estimation framework utilizing deep learning architectures. Plant images captured from different angles were fused to estimate the yield and subsequently to ra…
RepSFNet : A Single Fusion Network with Structural Reparameterization for Crowd Counting
Department of Electro-Optical Engineering（电子光学工程系）；National Formosa University（国家Formosa大学）；Department of Electrical Engineering（电气工程系）；National Taipei University（国家台北大学）；College of Artificial Intelligence and Green Energy（人工智能与绿色能源学院）；National Yang Ming Chiao Tung University（国家阳明交通大学）
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡大学模式识别实验室）；Institute of Veterinary Pathology, Free University Berlin（柏林自由大学兽医病理学研究所）
Identification and counting of cells and mitotic figures is a standard task in diagnostic histopathology. Due to the large overall cell count on histological slides and the potential sparse prevalence of some relevant cell types or mitotic figures, retrieving annotation data for sufficient statistics is a tedious task and prone to a significant error in assessment. Automatic classification and segmentation is a classic task in digital pathology, yet it is not solved to a sufficient degree. We present a novel approach for cell and mitotic figure classification, based on a deep convolutional net…
The Harker School（哈克学校）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）
Current analysis of tumor proliferation, the most salient prognostic biomarker for invasive breast cancer, is limited to subjective mitosis counting by pathologists in localized regions of tissue images. This study presents the first data-driven integrative approach to characterize the severity of tumor growth and spread on a categorical and molecular level, utilizing multiple biologically salient deep learning classifiers to develop a comprehensive prognostic model. Our approach achieves pathologist-level performance on three-class categorical tumor severity prediction. It additionally pionee…
Duke University（杜克大学）；Institute of North Electronic Equipment（北方电子设备研究所）
Crowd counting usually addressed by density estimation becomes an increasingly important topic in computer vision due to its widespread applications in video surveillance, urban planning, and intelligence gathering. However, it is essentially a challenging task because of the greatly varied sizes of objects, coupled with severe occlusions and vague appearance of extremely small individuals. Existing methods heavily rely on multi-column learning architectures to extract multi-scale features, which however suffer from heavy computational cost, especially undesired for crowd counting. In this pap…
Image segmentation for video analysis plays an essential role in different research fields such as smart city, healthcare, computer vision and geoscience, and remote sensing applications. In this regard, a significant effort has been devoted recently to developing novel segmentation strategies; one of the latest outstanding achievements is panoptic segmentation. The latter has resulted from the fusion of semantic and instance segmentation. Explicitly, panoptic segmentation is currently under study to help gain a more nuanced knowledge of the image scenes for video surveillance, crowd counting,…
Thermal-Only Crowd Counting with Deployment-Time Privacy Protection
School of Computer Science, University of Nottingham（诺丁汉大学计算机科学学院）；School of Computer Science, University of St Andrews（斯特灵大学计算机科学学院）；Department of Data Science, City University of Hong Kong（香港城市大学数据科学系）；Harbin Institute of Technology（哈尔滨工业大学）
In this paper, we attempt to address the challenging problem of counting built-structures in the satellite imagery. Building density is a more accurate estimate of the population density, urban area expansion and its impact on the environment, than the built-up area segmentation. However, building shape variances, overlapping boundaries, and variant densities make this a complex task. To tackle this difficult problem, we propose a deep learning based regression technique for counting built-structures in satellite imagery. Our proposed framework intelligently combines features from different re…
The Effectiveness of a Simplified Model Structure for Crowd Counting
school of Opto-Electronic and Communication Engineering, Xiamen University of Technology（光学电子与通信工程学院，厦门理工大学）；department of Artificial Intelligence, School of Informatics, Xiamen University（人工智能系，厦门大学）；department of Computer Science, Institute of Mathematics, Physics and Computer Science, Aberystwyth University（计算机科学系，数学、物理与计算机科学学院，阿伯里斯特维斯大学）；department of Electrical Engineering, Yuan Ze University（电子工程系，元智大学）
Baidu Research（百度研究院）；German Aerospace Center（德国航空航天中心）；City University of Hong Kong（香港城市大学）；Northwestern Polytechnical University（西北工业大学）；Technical University of Munich（慕尼黑工业大学）
Visual crowd counting has been recently studied as a way to enable people counting in crowd scenes from images. Albeit successful, vision-based crowd counting approaches could fail to capture informative features in extreme conditions, e.g., imaging at night and occlusion. In this work, we introduce a novel task of audiovisual crowd counting, in which visual and auditory information are integrated for counting purposes. We collect a large-scale benchmark, named auDiovISual Crowd cOunting (DISCO) dataset, consisting of 1,935 images and the corresponding audio clips, and 170,270 annotated instan…
Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Facebook AI Research（Facebook人工智能研究院）
Machine learning models tend to over-rely on statistical shortcuts. These spurious correlations between parts of the input and the output labels does not hold in real-world settings. We target this issue on the recent open-ended visual counting task which is well suited to study statistical shortcuts. We aim to develop models that learn a proper mechanism of counting regardless of the output label. First, we propose the Modifying Count Distribution (MCD) protocol, which penalizes models that over-rely on statistical shortcuts. It is based on pairs of training and testing sets that do not follo…
Beijing University of Technology（北京工业大学）；China Agricultural University（中国农业大学）；JD Finance America Corporation（京东金融美国公司）
Recently the crowd counting has received more and more attention. Especially the technology of high-density environment has become an important research content, and the relevant methods for the existence of extremely dense crowd are not optimal. In this paper, we propose a multi-level attentive Convolutional Neural Network (MLAttnCNN) for crowd counting. We extract high-level contextual information with multiple different scales applied in pooling, and use multi-level attention modules to enrich the characteristics at different layers to achieve more efficient multi-scale feature fusion, whic…
Real-Time Crowd Counting for Embedded Systems with Lightweight Architecture
Institute of Artificial Intelligence (TeleAI)（人工智能研究所（TeleAI））；China Telecom（中国电信）；School of Artifcial Intelligence, Optics and Electronics (iOPEN)（人工智能、光学与电子学学院（iOPEN））；Northwestern Polytechnical University（西北工业大学）；Honors College（荣誉学院）
In recent years, significant progress has been made on the research of crowd counting. However, as the challenging scale variations and complex scenes existed in crowds, neither traditional convolution networks nor recent Transformer architectures with fixed-size attention could handle the task well. To address this problem, this paper proposes a scene-adaptive attention network, termed SAANet. First of all, we design a deformable attention in-built Transformer backbone, which learns adaptive feature representations with deformable sampling locations and dynamic attention weights. Then we prop…
Beihang University（北京航空航天大学）；Inception Institute of Artificial Intelligence（Inception人工智能研究院）；University at Buffalo（布法罗大学）
Crowd counting has recently attracted increasing interest in computer vision but remains a challenging problem. In this paper, we propose a trellis encoder-decoder network (TEDnet) for crowd counting, which focuses on generating high-quality density estimation maps. The major contributions are four-fold. First, we develop a new trellis architecture that incorporates multiple decoding paths to hierarchically aggregate features at different encoding stages, which can handle large variations of objects. Second, we design dense skip connections interleaved across paths to facilitate sufficient mul…
Chongqing University（重庆大学）；Huazhong University of Science and Technology（华中科技大学）
The existing crowd counting methods usually adopted attention mechanism to tackle background noise, or applied multi-level features or multi-scales context fusion to tackle scale variation. However, these approaches deal with these two problems separately. In this paper, we propose a Hybrid Attention Network (HAN) by employing Progressive Embedding Scale-context (PES) information, which enables the network to simultaneously suppress noise and adapt head scale variation. We build the hybrid attention mechanism through paralleling spatial attention and channel attention module, which makes the n…
Recently, much advance has been made in image captioning, and an encoder-decoder framework has been adopted by all the state-of-the-art models. Under this framework, an input image is encoded by a convolutional neural network (CNN) and then translated into natural language with a recurrent neural network (RNN). The existing models counting on this framework merely employ one kind of CNNs, e.g., ResNet or Inception-X, which describe image contents from only one specific view point. Thus, the semantic meaning of an input image cannot be comprehensively understood, which restricts the performance…
