Automating the analysis of imagery of the Gastrointestinal (GI) tract captured during endoscopy procedures has substantial potential benefits for patients, as it can provide diagnostic support to medical practitioners and reduce mistakes via human error. To further the development of such methods, we propose a two-stream model for endoscopic image analysis. Our model fuses two streams of deep feature inputs by mapping their inherent relations through a novel relational network model, to better model symptoms and classify the image. In contrast to handcrafted feature-based models, our proposed network is able to learn features automatically and outperforms existing state-of-the-art methods on two public datasets: KVASIR and Nerthus. Our extensive evaluations illustrate the importance of having two streams of inputs instead of a single stream and also demonstrates the merits of the proposed relational network architecture to combine those streams.
University of Zaragoza（萨拉戈萨大学）；University College London（伦敦大学学院）
There is often a significant gap between research results and applicability in routine medical practice. This work studies the performance of well-known local features on a medical dataset captured during routine colonoscopy procedures. Local feature extraction and matching is a key step for many computer vision applications, specially regarding 3D modelling. In the medical domain, handcrafted local features such as SIFT, with public pipelines such as COLMAP, are still a predominant tool for this kind of tasks. We explore the potential of the well known self-supervised approach SuperPoint, pre…
Training-Time Optical Priors for Wireless Capsule Endoscopy Classification: Hemoglobin-Aware Input Fusion with Cross-Vendor Evaluation
Department of Biomedical Engineering, University of Texas Southwestern Medical Center（生物医学工程系，德克萨斯西南医学中心）；Department of Internal Medicine, University of Texas Southwestern Medical Center（内科系，德克萨斯西南医学中心）；Department of Radiation Oncology, Stanford University（放射肿瘤学系，斯坦福大学）；VELVETECH, LLC（VELVETECH公司）；Division of Digestive and Liver Diseases, Clements University Hospital（消化和肝病系，克莱门斯大学医院）；Internal Medicine, Division of Digestive and Liver Diseases, Parkland Hospital（内科，消化和肝病系，帕克兰医院）
Rethinking Exemplars for Continual Semantic Segmentation in Endoscopy Scenes: Entropy-based Mini-Batch Pseudo-Replay
The Chinese University of Hong Kong（香港中文大学）；Northeastern University（东北大学）；The University of Sydney（悉尼大学）；National University of Singapore（新加坡国立大学）；Suzhou Research Institute, National University of Singapore（新加坡国立大学苏州研究院）；Shun Hing Institute of Advanced Engineering, The Chinese University of Hong Kong（香港中文大学信兴高等工程研究院）；Shenzhen Research Institute, The Chinese University of Hong Kong（香港中文大学深圳研究院）
EndoVLM: An Endoscopy Vision-Language Pre-training Model via Anatomy-Guided Sparsity and Progressive Alignment
DAMO Academy, Alibaba Group（阿里巴巴达摩院）；The First Affiliated Hospital of Zhejiang Chinese Medical University（浙江中医药大学附属第一医院）；Shanghai Jiao Tong University（上海交通大学）；Hupan Lab（湖畔实验室）；Zhejiang University（浙江大学）
Video streams are utilised to guide minimally-invasive surgery and diagnostic procedures in a wide range of procedures, and many computer assisted techniques have been developed to automatically analyse them. These approaches can provide additional information to the surgeon such as lesion detection, instrument navigation, or anatomy 3D shape modeling. However, the necessary image features to recognise these patterns are not always reliably detected due to the presence of irregular light patterns such as specular highlight reflections. In this paper, we aim at removing specular highlights from…
Smart Engines Service LLC（智能引擎服务有限责任公司）；FRC CSC RAS（俄罗斯科学院联邦研究中心计算机科学与控制研究所）；MIPT（莫斯科物理技术学院）
In this paper, we consider a task of stopping the video stream recognition process of a text field, in which each frame is recognized independently and the individual results are combined together. The video stream recognition stopping problem is an under-researched topic with regards to computer vision, but its relevance for building high-performance video recognition systems is clear. Firstly, we describe an existing method of optimally stopping such a process based on a modelling of the next combined result. Then, we describe approximations and assumptions which allowed us to build an optim…
North University of China（中北大学）；Shandong University（山东大学）；University of Electronic Science and Technology of China（电子科技大学）；University of Wollongong（伍伦贡大学）
Pose based hand gesture recognition has been widely studied in the recent years. Compared with full body action recognition, hand gesture involves joints that are more spatially closely distributed with stronger collaboration. This nature requires a different approach from action recognition to capturing the complex spatial features. Many gesture categories, such as "Grab" and "Pinch", have very similar motion or temporal patterns posing a challenge on temporal processing. To address these challenges, this paper proposes a two-stream neural network with one stream being a self-attention based …
University of Arizona（亚利桑那大学）；Ohio State University（俄亥俄州立大学）
Gesture recognition and hand motion tracking are important tasks in advanced gesture based interaction systems. In this paper, we propose to apply a sliding windows filtering approach to sample the incoming streams of data from data gloves and a decision tree model to recognize the gestures in real time for a manual grafting operation of a vegetable seedling propagation facility. The sequence of these recognized gestures defines the tasks that are taking place, which helps to evaluate individuals' performances and to identify any bottlenecks in real time. In this work, two pairs of data gloves…
Moscow Institute of Physics and Technology “MIPT”（莫斯科物理技术学院）；Smart Engines（智能引擎公司）；Federal Research Center “Computer Science and Control” of Russian Academy of Sciences（俄罗斯科学院计算机科学与控制联邦研究中心）
In the field of document analysis and recognition using mobile devices for capturing, and the field of object recognition in a video stream, an important problem is determining the time when the capturing process should be stopped. Efficient stopping influences not only the total time spent for performing recognition and data entry, but the expected accuracy of the result as well. This paper is directed on extending the stopping method based on next integrated recognition result modelling, in order for it to be used within a string result recognition model with per-character alternatives. The …
Mercedes-Benz R&D India（梅赛德斯-奔驰印度研发中心）；Indian Institute of Technology, Bombay（印度理工学院孟买分校）；Axogyan AI
Online and Early detection of gestures is crucial for building touchless gesture based interfaces. These interfaces should operate on a stream of video frames instead of the complete video and detect the presence of gestures at an earlier stage than post-completion for providing real time user experience. To achieve this, it is important to recognize the progression of the gesture across different stages so that appropriate responses can be triggered on reaching the desired execution stage. To address this, we propose a simple yet effective multi-task learning framework which models the progre…
Where data is available, it is desirable in geostatistical modelling to make use of additional covariates, for example terrain data, in order to improve prediction accuracy in the modelling task. While elevation itself may be important, additional explanatory power for any given problem can be sought (but not necessarily found) by filtering digital elevation models to extract higher-order derivatives such as slope angles, curvatures, and roughness. In essence, it would be beneficial to extract as much task-relevant information as possible from the elevation grid. However, given the complexitie…
Dual-stream attention-guided learning for weakly supervised whole slide image classification
College of Computer Science and Technology, College of Data Science, Taiyuan University of Technology（太原科技大学计算机科学与技术学院、数据科学学院）；College of Humanities, Law and Foreign Languages, Taiyuan University of Technology（太原科技大学人文学院、法律与外语学院）；College of Artificial Intelligence, Taiyuan University of Technology（太原科技大学人工智能学院）；School of Cyberspace Security, Beijing University of Posts and Telecommunications（北京邮电大学网络安全学院）；School of Mathematics, Taiyuan University of Technology（太原科技大学数学学院）
