Pair-wise loss is an approach to metric learning that learns a semantic embedding by optimizing a loss function that encourages images from the same semantic class to be mapped closer than images from different classes. The literature reports a large and growing set of variations of the pair-wise loss strategies. Here we decompose the gradient of these loss functions into components that relate to how they push the relative feature positions of the anchor-positive and anchor-negative pairs. This decomposition allows the unification of a large collection of current pair-wise loss functions. Additionally, explicitly constructing pair-wise gradient updates to separate out these effects gives insights into which have the biggest impact, and leads to a simple algorithm that beats the state of the art for image retrieval on the CAR, CUB and Stanford Online products datasets.
Dissecting Representation Misalignment in Contrastive Learning via Influence Function
Provable Responsible AI and Data Analytics (PRADA) Lab（可解释负责任AI与数据分析（PRADA）实验室）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Shanghai Jiao Tong University（上海交通大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）；The University of Auckland（奥克兰大学）；RIKEN Center for Advanced Intelligence Project (AIP)（理化学研究所高级智能项目中心）；Arizona State University（亚利桑那州立大学）
University of Salzburg（萨尔茨堡大学）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）
Minimizing cross-entropy over the softmax scores of a linear map composed with a high-capacity encoder is arguably the most popular choice for training neural networks on supervised learning tasks. However, recent works show that one can directly optimize the encoder instead, to obtain equally (or even more) discriminative representations via a supervised variant of a contrastive objective. In this work, we address the question whether there are fundamental differences in the sought-for representation geometry in the output space of the encoder at minimal loss. Specifically, we prove, under mi…
Visual-Semantic Embedding (VSE) is a prevalent approach in image-text retrieval by learning a joint embedding space between the image and language modalities where semantic similarities would be preserved. The triplet loss with hard-negative mining has become the de-facto objective for most VSE methods. Inspired by recent progress in deep metric learning (DML) in the image domain which gives rise to new loss functions that outperform triplet loss, in this paper, we revisit the problem of finding better objectives for VSE in image-text matching. Despite some attempts in designing losses based o…
Beyond scalar losses: calibrating segmentation models via gradient vector field surgery
School of Computation, Information and Technology, TUM（慕尼黑工业大学计算、信息与技术学院）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Department of Radiology, Weill Cornell Medicine（威尔康乃尔医学院放射科）；School of Medicine and Health, TUM University Hospital（慕尼黑工业大学医院医学与健康学院）；Cornell Tech（康奈尔科技学院）；Department of Computing, Imperial College London（伦敦帝国理工学院计算系）
NUS Graduate School for Integrative Sciences and Engineering (NGS)（新加坡国立大学综合科学与工程研究生院（NGS））；National University of Singapore（新加坡国立大学）；University of Moratuwa（莫拉图瓦大学）；NIT Trichy（印度国立理工学院蒂鲁吉拉伯利分校）
Robot-assisted surgery is an emerging technology which has undergone rapid growth with the development of robotics and imaging systems. Innovations in vision, haptics and accurate movements of robot arms have enabled surgeons to perform precise minimally invasive surgeries. Real-time semantic segmentation of the robotic instruments and tissues is a crucial step in robot-assisted surgery. Accurate and efficient segmentation of the surgical scene not only aids in the identification and tracking of instruments but also provided contextual information about the different tissues and instruments be…
Anatomy-Informed Neural Networks: Encoding Anatomic Priors in Loss and Architecture, with an SE(3) Formulation of Guidewire-Induced Aortoiliac Deformation
Indian Institute of Technology, Kharagpur（印度理工学院卡拉格普尔分校）
Surgical workflow analysis is of importance for understanding onset and persistence of surgical phases and individual tool usage across surgery and in each phase. It is beneficial for clinical quality control and to hospital administrators for understanding surgery planning. Video acquired during surgery typically can be leveraged for this task. Currently, a combination of convolutional neural network (CNN) and recurrent neural networks (RNN) are popularly used for video analysis in general, not only being restricted to surgical videos. In this paper, we propose a multi-task learning framework…
NUS Graduate School for Integrative Sciences and Engineering (NGS), National University of Singapore（新加坡国立大学NUS综合科学与工程研究生院（NGS））；National University of Singapore（新加坡国立大学）；University of Michigan-Joint Institute, Shanghai Jiao Tong University（上海交通大学密歇根联合学院）
Directing of the task-specific attention while tracking instrument in surgery holds great potential in robot-assisted intervention. For this purpose, we propose an end-to-end trainable multitask learning (MTL) model for real-time surgical instrument segmentation and attention prediction. Our model is designed with a weight-shared encoder and two task-oriented decoders and optimized for the joint tasks. We introduce batch-Wasserstein (bW) loss and construct a soft attention module to refine the distinctive visual region for efficient saliency learning. For multitask optimization, it is always c…
Semantic segmentation in cataract surgery has a wide range of applications contributing to surgical outcome enhancement and clinical risk reduction. However, the varying issues in segmenting the different relevant instances make the designation of a unique network quite challenging. This paper proposes a semantic segmentation network termed as DeepPyram that can achieve superior performance in segmenting relevant objects in cataract surgery videos with varying issues. This superiority mainly originates from three modules: (i) Pyramid View Fusion, which provides a varying-angle global view of t…
Dynamic Robot-Assisted Surgery with Hierarchical Class-Incremental Semantic Segmentation
Department of Computer Science, University of Freiburg（弗赖堡大学计算机科学系）；Zuse School ELIZA（Zuse学校ELIZA）；NeuroPoly Lab, Institute of Biomedical Engineering, Polytechnique Montréal（神经Poly实验室，生物医学工程学院，蒙特利尔理工学院）；Mila - Québec AI Institute（魁北克人工智能研究所）；Collaborative Research Institute Intelligent Oncology (CRIION)（智能肿瘤学协作研究机构）
University of Maryland（马里兰大学）；The Hong Kong Polytechnic University（香港理工大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
In this paper, we leverage the recent advances in physics-informed neural network (PINN) and develop a generic PINN-based framework to assess the reliability of multi-state systems (MSSs). The proposed methodology consists of two major steps. In the first step, we recast the reliability assessment of MSS as a machine learning problem using the framework of PINN. A feedforward neural network with two individual loss groups are constructed to encode the initial condition and state transitions governed by ordinary differential equations (ODEs) in MSS. Next, we tackle the problem of high imbalance…
University College London (UCL)（伦敦大学学院）；King’s College London (KCL)（伦敦国王学院）；Moorfields Eye Hospital（摩尔菲尔兹眼科医院）；Institute of Ophthalmology, University College London（伦敦大学学院眼科研究所）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences（惠康/英国工程与物理科学研究委员会介入与外科科学中心）；School of Biomedical Engineering & Imaging Sciences（生物医学工程与影像科学学院）
Our work proposes neural network design choices that set the state-of-the-art on a challenging public benchmark on cataract surgery, CaDIS. Our methodology achieves strong performance across three semantic segmentation tasks with increasingly granular surgical tool class sets by effectively handling class imbalance, an inherent challenge in any surgical video. We consider and evaluate two conceptually simple data oversampling methods as well as different loss functions. We show significant performance gains across network architectures and tasks especially on the rarest tool classes, thereby p…
GSMorph: Gradient Surgery for cine-MRI Cardiac Deformable Registration
University of Leeds（利兹大学）；Radboud University Medical Centre（拉德堡德大学医学中心）；Netherlands Cancer Institute（荷兰癌症研究所）；Shenzhen University（深圳大学）；Shenzhen RayShape Medical Technology Co., Ltd（深圳睿心医疗科技有限公司）
Vivekanand Education Society’s Institute of Technology（维韦卡南达教育协会技术学院）
Although modern face verification systems are accessible and accurate, they are not always robust to pose variance and occlusions. Moreover, accurate models require a large amount of data to train. We structure our experiments to operate on small amounts of data obtained from an NGO that funds ophthalmic surgeries. We set up our face verification task as that of verifying pre-operation and post-operation images of a patient that undergoes ophthalmic surgery, and as such the post-operation images have occlusions like an eye patch. In this paper, we present a system that performs the face verifi…
Two-View Accumulation as the Primary Training Lever for Hybrid-Capture Gaussian Splatting: A Variance-Decomposition View of When Gradient Surgery Helps
The Hong Kong University of Science and Technology（香港科技大学）
University of Bern（伯尔尼大学）；Alpen-Adria-Universität Klagenfurt（克拉根福阿尔卑斯-亚德里亚大学）
Semantic segmentation in cataract surgery has a wide range of applications contributing to surgical outcome enhancement and clinical risk reduction. However, the varying issues in segmenting the different relevant structures in these surgeries make the designation of a unique network quite challenging. This paper proposes a semantic segmentation network, termed DeepPyramid, that can deal with these challenges using three novelties: (1) a Pyramid View Fusion module which provides a varying-angle global view of the surrounding region centering at each pixel position in the input convolutional fe…
Rethinking Exemplars for Continual Semantic Segmentation in Endoscopy Scenes: Entropy-based Mini-Batch Pseudo-Replay
The Chinese University of Hong Kong（香港中文大学）；Northeastern University（东北大学）；The University of Sydney（悉尼大学）；National University of Singapore（新加坡国立大学）；Suzhou Research Institute, National University of Singapore（新加坡国立大学苏州研究院）；Shun Hing Institute of Advanced Engineering, The Chinese University of Hong Kong（香港中文大学信兴高等工程研究院）；Shenzhen Research Institute, The Chinese University of Hong Kong（香港中文大学深圳研究院）
Topology-Constrained Learning for Efficient Laparoscopic Liver Landmark Detection
The Hong Kong Polytechnic University, Hong Kong, China（香港理工大学）；National University of Singapore, Singapore（新加坡国立大学）；The Chinese University of Hong Kong, Hong Kong, China（香港中文大学）；Nanfang Hospital, Southern Medical University, Guangzhou, China（南方医科大学附属南华医院）
Object segmentation plays an important role in the modern medical image analysis, which benefits clinical study, disease diagnosis, and surgery planning. Given the various modalities of medical images, the automated or semi-automated segmentation approaches have been used to identify and parse organs, bones, tumors, and other regions-of-interest (ROI). However, these contemporary segmentation approaches tend to fail to predict the boundary areas of ROI, because of the fuzzy appearance contrast caused during the imaging procedure. To further improve the segmentation quality of boundary areas, w…
Zhejiang University（浙江大学）；University of Notre Dame（圣母大学）
Surgery is the only viable treatment for cataract patients with visual acuity (VA) impairment. Clinically, to assess the necessity of cataract surgery, accurately predicting postoperative VA before surgery by analyzing multi-view optical coherence tomography (OCT) images is crucially needed. Unfortunately, due to complicated fundus conditions, determining postoperative VA remains difficult for medical experts. Deep learning methods for this problem were developed in recent years. Although effective, these methods still face several issues, such as not efficiently exploring potential relations …
Simon Fraser University（西蒙弗雷泽大学）；Siemens Healthineers（西门子医疗）
Simultaneous segmentation of multiple organs from different medical imaging modalities is a crucial task as it can be utilized for computer-aided diagnosis, computer-assisted surgery, and therapy planning. Thanks to the recent advances in deep learning, several deep neural networks for medical image segmentation have been introduced successfully for this purpose. In this paper, we focus on learning a deep multi-organ segmentation network that labels voxels. In particular, we examine the critical choice of a loss function in order to handle the notorious imbalance problem that plagues both the …
Norwegian University of Science and Technology（挪威科技大学）
In minimally invasive surgery, the use of tissue dissection tools causes smoke, which inevitably degrades the image quality. This could reduce the visibility of the operation field for surgeons and introduces errors for the computer vision algorithms used in surgical navigation systems. In this paper, we propose a novel approach for computational smoke removal using supervised image-to-image translation. We demonstrate that straightforward application of existing generative algorithms allows removing smoke but decreases image quality and introduces synthetic noise (grid-structure). Thus, we pr…
The Hong Kong Polytechnic University（香港理工大学）；University of Oxford（牛津大学）
Automatic medical image segmentation based on Computed Tomography (CT) has been widely applied for computer-aided surgery as a prerequisite. With the development of deep learning technologies, deep convolutional neural networks (DCNNs) have shown robust performance in automated semantic segmentation of medical images. However, semantic segmentation algorithms based on DCNNs still meet the challenges of feature loss between encoder and decoder, multi-scale object, restricted field of view of filters, and lack of medical image data. This paper proposes a novel algorithm for automated vertebrae s…
Adaptive Confidence-Wise Loss for Improved Lens Structure Segmentation in AS-OCT
Research Institute of Trustworthy Autonomous Systems（可信自主系统研究 institutes）；Department of Computer Science（计算机科学系）；Engineering, Southern University of Science（科学大学工程学院）；Tomey Corporation, Nagoya 4510051, Japan（日本名古屋市Tomey公司）；Department of Electronic（电子系）；Information Engineering, Changchun University, Changchun 130022, China（长春大学信息工程系）；School of Computer Science, University of Nottingham Ningbo China, Ningbo 315100, China（诺丁汉大学宁波校区计算机科学学院）
Zhejiang Chinese Medical University（浙江中医药大学）；Royal Brompton Hospital（皇家布朗普顿医院）；Imperial College London（帝国理工学院）；University of California at Los Angeles（加州大学洛杉矶分校）
Glioma is the most common and aggressive brain tumor. Magnetic resonance imaging (MRI) plays a vital role to evaluate tumors for the arrangement of tumor surgery and the treatment of subsequent procedures. However, the manual segmentation of the MRI image is strenuous, which limits its clinical application. With the development of deep learning, a large number of automatic segmentation methods have been developed, but most of them stay in 2D images, which leads to subpar performance. Moreover, the serious voxel imbalance between the brain tumor and the background as well as the different sizes…
