With the increasing number of online learning material in the web, search for specific content in lecture videos can be time consuming. Therefore, automatic slide extraction from the lecture videos can be helpful to give a brief overview of the main content and to support the students in their studies. For this task, we propose a deep learning method to detect slide transitions in lectures videos. We first process each frame of the video by a heuristic-based approach using a 2-D convolutional neural network to predict transition candidates. Then, we increase the complexity by employing two 3-D convolutional neural networks to refine the transition candidates. Evaluation results demonstrate the effectiveness of our method in finding slide transitions.
SlideChain: Semantic Provenance for Lecture Understanding via Blockchain Registration
Department of Computer Science Rensselaer Polytechnic Institute（计算机科学系罗切斯特理工学院）；Department of Biomedical Engineering Rensselaer Polytechnic Institute（生物医学工程系罗切斯特理工学院）
AI-Generated Lecture Slides for Improving Slide Element Detection and Retrieval
Indian Institute of Technology（印度理工学院）；Sardar Vallabhbhai National Institute of Technology（萨达尔·瓦拉巴希国家理工学院）；CVIT, International Institute of Information Technology（信息技术国际学院）
Lecture slide presentations, a sequence of pages that contain text and figures accompanied by speech, are constructed and presented carefully in order to optimally transfer knowledge to students. Previous studies in multimedia and psychology attribute the effectiveness of lecture presentations to their multimodal nature. As a step toward developing AI to aid in student learning as intelligent teacher assistants, we introduce the Multimodal Lecture Presentations dataset as a large-scale benchmark testing the capabilities of machine learning models in multimodal understanding of educational cont…
This paper presents a novel approach for lecture video indexing using a boosted deep convolutional neural network system. The indexing is performed by matching high quality slide images, for which text is either known or extracted, to lower resolution video frames with possible noise, perspective distortion, and occlusions. We propose a deep neural network integrated with a boosting framework composed of two sub-networks targeting feature extraction and similarity determination to perform the matching. The trained network is given as input a pair of slide image and a candidate video frame imag…
UC Santa Barbara（加州大学圣巴巴拉分校）；Microsoft Research（微软研究院）
Creating presentation materials requires complex multimodal reasoning skills to summarize key concepts and arrange them in a logical and visually pleasing manner. Can machines learn to emulate this laborious process? We present a novel task and approach for document-to-slide generation. Solving this involves document summarization, image and text retrieval, slide structure and layout prediction to arrange key elements in a form suitable for presentation. We propose a hierarchical sequence-to-sequence approach to tackle our task in an end-to-end manner. Our approach exploits the inherent struct…
School of ITEE, The University of Queensland, Australia（昆士兰大学工学院（ITEE））；Sullivan Nicolaides Pathology, Australia（Sullivan Nicolaides病理实验室（澳大利亚））
This work tackles the automatic fine-grained slide quality assessment problem for digitized direct smears test using the Gram staining protocol. Automatic quality assessment can provide useful information for the pathologists and the whole digital pathology workflow. For instance, if the system found a slide to have a low staining quality, it could send a request to the automatic slide preparation system to remake the slide. If the system detects severe damage in the slides, it could notify the experts that manual microscope reading may be required. In order to address the quality assessment p…
MaViLS, a Benchmark Dataset for Video-to-Slide Alignment, Assessing Baseline Accuracy with a Multimodal Alignment Algorithm Leveraging Speech, OCR, and Visual Features
Karlsruhe University of Applied Sciences（卡尔斯鲁厄应用技术大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；University of Hohenheim（霍恩海姆大学）
Digital pathology enables remote access or consults and powerful image analysis algorithms. However, the slide digitization process can create artifacts such as out-of-focus (OOF). OOF is often only detected upon careful review, potentially causing rescanning and workflow delays. Although scan-time operator screening for whole-slide OOF is feasible, manual screening for OOF affecting only parts of a slide is impractical. We developed a convolutional neural network (ConvFocus) to exhaustively localize and quantify the severity of OOF regions on digitized slides. ConvFocus was developed using ou…
Over the last decade, online lecture videos have become increasingly popular and have experienced a meteoric rise during the pandemic. However, video-language research has primarily focused on instructional videos or movies, and tools to help students navigate the growing online lectures are lacking. Our first contribution is to facilitate research in the educational domain, by introducing AVLectures, a large-scale dataset consisting of 86 courses with over 2,350 lectures covering various STEM subjects. Each course contains video lectures, transcripts, OCR outputs for lecture frames, and optio…
University of Strasbourg（斯特拉斯堡大学）；INSA-Lyon（法国国立应用科学学院里昂分校）；Université Claude Bernard Lyon 1（克洛德·贝尔纳里昂第一大学）；Hannover Medical School（汉诺威医学院）；Université de Haute Alsace（上阿尔萨斯大学）；University Clinic, Freiburg（弗莱堡大学医院）
Objective: This article presents an automatic image processing framework to extract quantitative high-level information describing the micro-environment of glomeruli in consecutive whole slide images (WSIs) processed with different staining modalities of patients with chronic kidney rejection after kidney transplantation. Methods: This four-step framework consists of: 1) approximate rigid registration, 2) cell and anatomical structure segmentation 3) fusion of information from different stainings using a newly developed registration algorithm 4) feature extraction. Results: Each step of the fr…
Joint analysis of multiple biomarker images and tissue morphology is important for disease diagnosis, treatment planning and drug development. It requires cross-staining comparison among Whole Slide Images (WSIs) of immuno-histochemical and hematoxylin and eosin (H&E) microscopic slides. However, automatic, and fast cross-staining alignment of enormous gigapixel WSIs at single-cell precision is challenging. In addition to morphological deformations introduced during slide preparation, there are large variations in cell appearance and tissue morphology across different staining. In this paper, …
Dartmouth-Hitchcock Medical Center（达特茅斯-希区柯克医疗中心）；Dartmouth College（达特茅斯学院）
Classification of histologic patterns in lung adenocarcinoma is critical for determining tumor grade and treatment for patients. However, this task is often challenging due to the heterogeneous nature of lung adenocarcinoma and the subjective criteria for evaluation. In this study, we propose a deep learning model that automatically classifies the histologic patterns of lung adenocarcinoma on surgical resection slides. Our model uses a convolutional neural network to identify regions of neoplastic cells, then aggregates those classifications to infer predominant and minor histologic patterns f…
University of British Columbia（不列颠哥伦比亚大学）；British Columbia Cancer Research Centre（不列颠哥伦比亚省癌症研究中心）
This paper addresses the problem of quantifying biomarkers in multi-stained tissues, based on color and spatial information. A deep learning based method that can automatically localize and quantify the cells expressing biomarker(s) in a whole slide image is proposed. The deep learning network is a fully convolutional network (FCN) whose input is the true RGB color image of a tissue and output is a map of the different biomarkers. The FCN relies on a convolutional neural network (CNN) that classifies each cell separately according to the biomarker it expresses. In this study, images of immunoh…
We propose an end-to-end lecture video generation system that can generate realistic and complete lecture videos directly from annotated slides, instructor's reference voice and instructor's reference portrait video. Our system is primarily composed of a speech synthesis module with few-shot speaker adaptation and an adversarial learning-based talking-head generation module. It is capable of not only reducing instructors' workload but also changing the language and accent which can help the students follow the lecture more easily and enable a wider dissemination of lecture contents. Our experi…
University of Heidelberg（海德堡大学）；DKFZ Heidelberg（海德堡德国癌症研究中心）
The progression of breast cancer can be quantified in lymph node whole-slide images (WSIs). We describe a novel method for effectively performing classification of whole-slide images and patient level breast cancer grading. Our method utilises a deep neural network. The method performs classification on small patches and uses model averaging for boosting. In the first step, region of interest patches are determined and cropped automatically by color thresholding and then classified by the deep neural network. The classification results are used to determine a slide level class and for further …
Convolutional Neural Networks (CNN) are state-of-the-art models for many image classification tasks. However, to recognize cancer subtypes automatically, training a CNN on gigapixel resolution Whole Slide Tissue Images (WSI) is currently computationally impossible. The differentiation of cancer subtypes is based on cellular-level visual features observed on image patch scale. Therefore, we argue that in this situation, training a patch-level classifier on image patches will perform better than or similar to an image-level classifier. The challenge becomes how to intelligently combine patch-lev…
EndoNet: model for automatic calculation of H-score on histological slides
Research Institute of Molecular and Cellular Medicine, Peoples’ Friendship University of Russia (RUDN University)（俄罗斯人民友谊大学分子与细胞医学研究所）；A.P. Avtsyn Scientific Research Institute of Human Morphology（阿夫钦人类形态学科学研究所）
Pancreatic cancer will soon be the second leading cause of cancer-related death in Western society. Imaging techniques such as CT, MRI and ultrasound typically help providing the initial diagnosis, but histopathological assessment is still the gold standard for final confirmation of disease presence and prognosis. In recent years machine learning approaches and pathomics pipelines have shown potential in improving diagnostics and prognostics in other cancerous entities, such as breast and prostate cancer. A crucial first step in these pipelines is typically identification and segmentation of t…
Throughout the world, breast cancer is one of the leading causes of female death. Recently, deep learning methods are developed to automatically grade breast cancer of histological slides. However, the performance of existing deep learning models is limited due to the lack of large annotated biomedical datasets. One promising way to relieve the annotating burden is to leverage the unannotated datasets to enhance the trained model. In this paper, we first apply active learning method in breast cancer grading, and propose a semi-supervised framework based on expectation maximization (EM) model. …
Metastatic presence in lymph nodes is one of the most important prognostic variables of breast cancer. The current diagnostic procedure for manually reviewing sentinel lymph nodes, however, is very time-consuming and subjective. Pathologists have to manually scan an entire digital whole-slide image (WSI) for regions of metastasis that are sometimes only detectable under high resolution or entirely hidden from the human visual cortex. From October 2015 to April 2016, the International Symposium on Biomedical Imaging (ISBI) held the Camelyon Grand Challenge 2016 to crowd-source ideas and algorit…
PreMind: Multi-Agent Video Understanding for Advanced Indexing of Presentation-style Videos
Bosch Research North America（博世北美研究院）；Bosch Center for Artificial Intelligence (BCAI)（博世人工智能中心）；Department of Computer Science and Engineering, Texas A&M University（德克萨斯农工大学计算机科学与工程学院）
Automatic Registration of SHG and H&E Images with Feature-based Initial Alignment and Intensity-based Instance Optimization: Contribution to the COMULIS Challenge
AGH University of Krakow（克拉科夫AGH科技大学）；University of Applied Sciences Western Switzerland (HES-SO Valais-Wallis)（瑞士西部应用科学大学（瓦莱州-瓦利斯））；University of Geneva（日内瓦大学）
Roche Innovation Center Munich（罗氏慕尼黑创新中心）；Technische Universität München（慕尼黑工业大学）；SagivTech LTD.（SagivTech有限公司）
A key challenge in cancer immunotherapy biomarker research is quantification of pattern changes in microscopic whole slide images of tumor biopsies. Different cell types tend to migrate into various tissue compartments and form variable distribution patterns. Drug development requires correlative analysis of various biomarkers in and between the tissue compartments. To enable that, tissue slides are manually annotated by expert pathologists. Manual annotation of tissue slides is a labor intensive, tedious and error-prone task. Automation of this annotation process can improve accuracy and cons…
The Chinese University of Hong Kong（香港中文大学）；Imsight Medical Technology Inc（宜远智能医疗科技有限公司）；Xiamen University（厦门大学）；The Hong Kong Polytechnic University（香港理工大学）
Lymph node metastasis is one of the most significant diagnostic indicators in breast cancer, which is traditionally observed under the microscope by pathologists. In recent years, computerized histology diagnosis has become one of the most rapidly expanding fields in medical image computing, which alleviates pathologists' workload and reduces misdiagnosis rate. However, automatic detection of lymph node metastases from whole slide images remains a challenging problem, due to the large-scale data with enormous resolutions and existence of hard mimics. In this paper, we propose a novel framework…
Geisel School of Medicine at Dartmouth（达特茅斯盖泽尔医学院）；Dartmouth-Hitchcock Medical Center（达特茅斯-希契科克医疗中心）；Dartmouth College（达特茅斯学院）
Renal cell carcinoma (RCC) is the most common renal cancer in adults. The histopathologic classification of RCC is essential for diagnosis, prognosis, and management of patients. Reorganization and classification of complex histologic patterns of RCC on biopsy and surgical resection slides under a microscope remains a heavily specialized, error-prone, and time-consuming task for pathologists. In this study, we developed a deep neural network model that can accurately classify digitized surgical resection slides and biopsy slides into five related classes: clear cell RCC, papillary RCC, chromop…
Generalisation of automatic tumour segmentation in histopathological whole-slide images across multiple cancer types
Institute for Cancer Genetics and Informatics（癌症基因组学与信息学研究所）；Oslo University Hospital（奥斯陆大学医院）；University of Oslo（奥斯陆大学）；Department of Gastrointestinal（胃肠科部）；Institute of Clinical Medicine（临床医学研究所）；Department of Gynaecological Oncology（妇科肿瘤科部）；Amsterdam University Medical Centres（阿姆斯特丹大学医学中心）；Department of Obstetrics and Gynaecology（妇产科部）；Comprehensive Cancer Center Innsbruck（因斯布鲁克综合癌症中心）；Innsbruck Medical University（因斯布鲁克医学院）；Vestre Viken Hospital Trust（维斯特维肯医院信托）；Department of Medical Biology（医学生物学部）；UiT The Arctic University of Norway（挪威北极大学（UiT））；Department of Clinical Pathology（临床病理学部）
Stevens Institute of Technology（史蒂文斯理工学院）；University of Luebeck（吕贝克大学）；Northeastern University（东北大学）；China Medical University（中国医科大学）
With the development of computer-aided diagnosis (CAD) and image scanning technology, Whole-slide Image (WSI) scanners are widely used in the field of pathological diagnosis. Therefore, WSI analysis has become the key to modern digital pathology. Since 2004, WSI has been used more and more in CAD. Since machine vision methods are usually based on semi-automatic or fully automatic computers, they are highly efficient and labor-saving. The combination of WSI and CAD technologies for segmentation, classification, and detection helps histopathologists obtain more stable and quantitative analysis r…
Slide presentations are an effective and efficient tool used by the teaching community for classroom communication. However, this teaching model can be challenging for blind and visually impaired (VI) students. The VI student required personal human assistance for understand the presented slide. This shortcoming motivates us to design a Classroom Slide Narration System (CSNS) that generates audio descriptions corresponding to the slide content. This problem poses as an image-to-markup language generation task. The initial step is to extract logical regions such as title, text, equation, figure…
College of Aerospace Science and Engineering, National University of Defense Technology（国防科技大学航空航天科学与工程学院）；Eindhoven University of Technology（埃因霍温理工大学）；ScreenPoint Medical（ScreenPoint Medical 公司）；Pingan Technology（平安科技）；Lunit Inc.（Lunit 公司）；Queen Mary University of London（伦敦玛丽女王大学）；Beihang University（北京航空航天大学）；Beijing Advanced Innovation Center for Biomedical Engineering（北京生物医学工程高精尖创新中心）
Accurate segmentation of lung cancer in pathology slides is a critical step in improving patient care. We proposed the ACDC@LungHP (Automatic Cancer Detection and Classification in Whole-slide Lung Histopathology) challenge for evaluating different computer-aided diagnosis (CADs) methods on the automatic diagnosis of lung cancer. The ACDC@LungHP 2019 focused on segmentation (pixel-wise detection) of cancer tissue in whole slide imaging (WSI), using an annotated dataset of 150 training images and 50 test images from 200 patients. This paper reviews this challenge and summarizes the top 10 submi…
Radboud University Medical Center（拉德堡德大学医学中心）；Canisius-Wilhelmina Hospital（卡尼修斯-威廉明娜医院）；Jeroen Bosch Hospital（耶伦博斯医院）
Manual counting of mitotic tumor cells in tissue sections constitutes one of the strongest prognostic markers for breast cancer. This procedure, however, is time-consuming and error-prone. We developed a method to automatically detect mitotic figures in breast cancer tissue sections based on convolutional neural networks (CNNs). Application of CNNs to hematoxylin and eosin (H&E) stained histological tissue sections is hampered by: (1) noisy and expensive reference standards established by pathologists, (2) lack of generalization due to staining variation across laboratories, and (3) high compu…
Cross-attention-based saliency inference for predicting cancer metastasis on whole slide images
Center for Biomedical Informatics, Wake Forest University School of Medicine（维克森林大学医学院生物医学信息学中心）；Department of Pathology, Wake Forest University School of Medicine（维克森林大学医学院病理学系）
INSA Centre Val de Loire（卢瓦尔河谷国立应用科学学院）；PRISME（PRISME实验室）
This paper addresses the problem of liver cancer segmentation in Whole Slide Image (WSI). We propose a multi-scale image processing method based on automatic end-to-end deep neural network algorithm for segmentation of cancer area. A seven-levels gaussian pyramid representation of the histopathological image was built to provide the texture information in different scales. In this work, several neural architectures were compared using the original image level for the training procedure. The proposed method is based on U-Net applied to seven levels of various resolutions (pyramidal subsumpling)…
Technische Hochschule Ingolstadt（英戈尔施塔特工业大学）；University Hospital Erlangen（埃尔朗根大学医院）；University of Veterinary Medicine Vienna（维也纳兽医大学）；Friedrich-Alexander-Universität Erlangen-Nürnberg（弗里德里希-亚历山大-埃尔朗根-纽伦堡大学）
Meningioma is one of the most prevalent brain tumors in adults. To determine its malignancy, it is graded by a pathologist into three grades according to WHO standards. This grade plays a decisive role in treatment, and yet may be subject to inter-rater discordance. In this work, we present and compare three approaches towards fully automatic meningioma grading from histology whole slide images. All approaches are following a two-stage paradigm, where we first identify a region of interest based on the detection of mitotic figures in the slide using a state-of-the-art object detection deep lea…
Lecture videos are an increasingly important learning resource for higher education. However, the challenge of quickly finding the content of interest in a lecture video is an important limitation of this format. This paper introduces visual summarization of lecture video segments to enhance navigation. A lecture video is divided into segments based on the frame-to-frame similarity of content. The user navigates the lecture video content by viewing a single frame visual and textual summary of each segment. The paper presents a novel methodology to generate the visual summary of a lecture video…
CourseBlueprint: A Structured Pipeline for Adaptive Pedagogical Video Generation Grounded in Course Corpora
Department of Computer Science, Rensselaer Polytechnic Institute（计算机科学系，拉特格斯理工学院）；Department of Biomedical Engineering, Rensselaer Polytechnic Institute（生物医学工程系，拉特格斯理工学院）
National Heart and Lung Institute, Imperial College London（伦敦帝国理工学院国家心肺研究所）；PingAn Healthcare Technology（平安医疗科技）；Technical University of Munich（慕尼黑工业大学）；National Clinical Research Center of Kidney Diseases, Nanjing University School of Medicine（南京大学医学院国家肾脏疾病临床医学研究中心）
Recognition of glomeruli lesions is the key for diagnosis and treatment planning in kidney pathology; however, the coexisting glomerular structures such as mesangial regions exacerbate the difficulties of this task. In this paper, we introduce a scheme to recognize fine-grained glomeruli lesions from whole slide images. First, a focal instance structural similarity loss is proposed to drive the model to locate all types of glomeruli precisely. Then an Uncertainty Aided Apportionment Network is designed to carry out the fine-grained visual classification without bounding-box annotations. This d…
Stain Style Transfer using Transitive Adversarial Networks
College of Physics and Information Engineering, Fuzhou University（福州大学物理与信息工程学院）；Fujian Key Lab of Medical Instrumentation & Pharmaceutical Technology（福建省医疗器械与制药技术重点实验室）；Imperial Vision Technology（英视科技）；Fujian Provincial Cancer Hospital, The Affiliated Hospital of Fujian Medical University（福建医科大学附属福建省肿瘤医院）
Histopathology relies on the analysis of microscopic tissue images to diagnose disease. A crucial part of tissue preparation is staining whereby a dye is used to make the salient tissue components more distinguishable. However, differences in laboratory protocols and scanning devices result in significant confounding appearance variation in the corresponding images. This variation increases both human error and the inter-rater variability, as well as hinders the performance of automatic or semi-automatic methods. In the present paper we introduce an unsupervised adversarial network to translat…
