We consider the task of representation learning for unsupervised segmentation of 3D voxel-grid biomedical images. We show that models that capture implicit hierarchical relationships between subvolumes are better suited for this task. To that end, we consider encoder-decoder architectures with a hyperbolic latent space, to explicitly capture hierarchical relationships present in subvolumes of the data. We propose utilizing a 3D hyperbolic variational autoencoder with a novel gyroplane convolutional layer to map from the embedding space back to 3D images. To capture these relationships, we introduce an essential self-supervised loss -- in addition to the standard VAE loss -- which infers approximate hierarchies and encourages implicitly related subvolumes to be mapped closer in the embedding space. We present experiments on both synthetic data and biomedical data to validate our hypothesis.
Massive collection and explosive growth of biomedical data, demands effective compression for efficient storage, transmission and sharing. Readily available visual data compression techniques have been studied extensively but tailored for natural images/videos, and thus show limited performance on biomedical data which are of different features and larger diversity. Emerging implicit neural representation (INR) is gaining momentum and demonstrates high promise for fitting diverse visual data in target-data-specific manner, but a general compression scheme covering diverse biomedical data is so…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, Chinese Academy of Sciences University（中国科学院大学人工智能学院）；National Laboratory of Pattern Recognition, CASIA（中国科学院自动化研究所模式识别国家重点实验室）；Center of Excellence in Brain Science and Intelligence Technology, CAS（中国科学院脑科学与智能技术卓越创新中心）
The continuity of biological tissue between consecutive biomedical images makes it possible for the video interpolation algorithm, to recover large area defects and tears that are common in biomedical images. However, noise and blur differences, large deformation, and drift between biomedical images, make the task challenging. To address the problem, this paper introduces a deformation-aware network to synthesize each pixel in accordance with the continuity of biological tissue. First, we develop a deformation-aware layer for consecutive biomedical images interpolation that implicitly adopting…
Camera-based implicit mind reading by capturing higher-order semantic dynamics of human gaze within environmental context
China University of Petroleum (East China)（中国石油大学（华东））；the Qingdao Institute of Software（青岛软件研究所）；the College of Computer Science and Technology（计算机科学与技术学院）；Shandong Key Laboratory of Intelligent Oil & Gas Industrial Software（山东省智能油气工业软件重点实验室）；College of Information Science and Technology（信息科学与技术学院）
VesselTok: Tokenizing Vessel-like 3D Biomedical Graph Representations for Reconstruction and Generation
University of Zurich, Switzerland（苏黎世大学）；ETH AI Center, Zurich, Switzerland（苏黎世联邦理工学院人工智能中心）；Technical University of Munich, Germany（慕尼黑技术大学）；Munich Center for Machine Learning (MCML), Germany（慕尼黑机器学习中心（MCML））
We present the first single-view 3D reconstruction network aimed at recovering geometric details from an input image which encompass both topological shape structures and surface features. Our key idea is to train the network to learn a detail disentangled reconstruction consisting of two functions, one implicit field representing the coarse 3D shape and the other capturing the details. Given an input image, our network, coined D$^2$IM-Net, encodes it into global and local features which are respectively fed into two decoders. The base decoder uses the global features to reconstruct a coarse i…
Cornell University（康奈尔大学）；Cornell Tech（康奈尔理工学院）；Massachusetts Institute of Technology（麻省理工学院）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
The convolution operation is a central building block of neural network architectures widely used in computer vision. The size of the convolution kernels determines both the expressiveness of convolutional neural networks (CNN), as well as the number of learnable parameters. Increasing the network capacity to capture rich pixel relationships requires increasing the number of learnable parameters, often leading to overfitting and/or lack of robustness. In this paper, we propose a powerful novel building block, the hyper-convolution, which implicitly represents the convolution kernel as a functi…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
The advancement of generative radiance fields has pushed the boundary of 3D-aware image synthesis. Motivated by the observation that a 3D object should look realistic from multiple viewpoints, these methods introduce a multi-view constraint as regularization to learn valid 3D radiance fields from 2D images. Despite the progress, they often fall short of capturing accurate 3D shapes due to the shape-color ambiguity, limiting their applicability in downstream tasks. In this work, we address this ambiguity by proposing a novel shading-guided generative implicit model that is able to learn a stark…
Predicting Genetic Mutation from Whole Slide Images via Biomedical-Linguistic Knowledge Enhanced Multi-label Classification
University of British Columbia（不列颠哥伦比亚大学）；Sun Yat-sen University Cancer Center（中山大学肿瘤防治中心）；Stanford University（斯坦福大学）；Australian Artificial Intelligence Institute, University of Technology Sydney（悉尼科技大学澳大利亚人工智能研究所）；School of Intelligent Systems Engineering, Sun Yat-sen University（中山大学智能工程学院）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）
Neural implicit functions have achieved impressive results for reconstructing 3D shapes from single images. However, the image features for describing 3D point samplings of implicit functions are less effective when significant variations of occlusions, views, and appearances exist from the image. To better encode image features, we study a geometry-aware convolutional kernel to leverage geometric relationships of point samplings by the proposed \emph{spatial pattern}, i.e., a structured point set. Specifically, the kernel operates at 2D projections of 3D points from the spatial pattern. Suppo…
U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation
Peter Munk Cardiac Centre, University Health Network（彼得·蒙克心脏中心，大学健康网络）；University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）
University of Michigan（密歇根大学）；Invenio Imaging（Invenio 成像公司）；New York University（纽约大学）
Learning high-quality, self-supervised, visual representations is essential to advance the role of computer vision in biomedical microscopy and clinical medicine. Previous work has focused on self-supervised representation learning (SSL) methods developed for instance discrimination and applied them directly to image patches, or fields-of-view, sampled from gigapixel whole-slide images (WSIs) used for cancer diagnosis. However, this strategy is limited because it (1) assumes patches from the same patient are independent, (2) neglects the patient-slide-patch hierarchy of clinical biomedical mic…
University of Tübingen（图宾根大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETAS GmbH, Bosch Group（博世集团ETAS有限公司）；Amazon（亚马逊）
Implicit representations of 3D objects have recently achieved impressive results on learning-based 3D reconstruction tasks. While existing works use simple texture models to represent object appearance, photo-realistic image synthesis requires reasoning about the complex interplay of light, geometry and surface properties. In this work, we propose a novel implicit representation for capturing the visual appearance of an object in terms of its surface light field. In contrast to existing representations, our implicit model represents surface light fields in a continuous fashion and independent …
Technical University of Munich（慕尼黑工业大学）；Meta Reality Labs Research（Meta现实实验室）
Parametric 3D models have formed a fundamental role in modeling deformable objects, such as human bodies, faces, and hands; however, the construction of such parametric models requires significant manual intervention and domain expertise. Recently, neural implicit 3D representations have shown great expressibility in capturing 3D shape geometry. We observe that deformable object motion is often semantically structured, and thus propose to learn Structured-implicit PArametric Models (SPAMs) as a deformable object representation that structurally decomposes non-rigid object motion into part-base…
Seoul National University（首尔国立大学）；Meta Reality Labs（Meta现实实验室）
To apply optical flow in practice, it is often necessary to resize the input to smaller dimensions in order to reduce computational costs. However, downsizing inputs makes the estimation more challenging because objects and motion ranges become smaller. Even though recent approaches have demonstrated high-quality flow estimation, they tend to fail to accurately model small objects and precise boundaries when the input resolution is lowered, restricting their applicability to high-resolution inputs. In this paper, we introduce AnyFlow, a robust network that estimates accurate flow from images o…
Self-supervised Learning of Neural Implicit Feature Fields for Camera Pose Refinement
Faculty of Electrical Engineering, Czech Technical University in Prague（布拉格捷克技术大学电气工程学院）；Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague（布拉格捷克技术大学捷克信息学、机器人学与控制论研究所）；NAVER LABS Europe（NAVER欧洲实验室）
University of Notre Dame（圣母大学）；Nanjing University of Science and Technology（南京理工大学）；Leibniz-Institut für Analytische Wissenschaften–ISAS–e.V.（莱布尼茨分析科学研究所）
Deep learning (DL) based semantic segmentation methods have achieved excellent performance in biomedical image segmentation, producing high quality probability maps to allow extraction of rich instance information to facilitate good instance segmentation. While numerous efforts were put into developing new DL semantic segmentation models, less attention was paid to a key issue of how to effectively explore their probability maps to attain the best possible instance segmentation. We observe that probability maps by DL semantic segmentation models can be used to generate many possible instance c…
Masaryk University（马萨里克大学）；University of Twente（特文特大学）
Methods allowing the synthesis of realistic cell shapes could help generate training data sets to improve cell tracking and segmentation in biomedical images. Deep generative models for cell shape synthesis require a light-weight and flexible representation of the cell shape. However, commonly used voxel-based representations are unsuitable for high-resolution shape synthesis, and polygon meshes have limitations when modeling topology changes such as cell growth or mitosis. In this work, we propose to use level sets of signed distance functions (SDFs) to represent cell shapes. We optimize a ne…
Capturing dynamical correlations using implicit neural representations
SLAC National Accelerator Laboratory（SLAC国家加速器实验室）；Department of Materials Science and Engineering, Stanford University（斯坦福大学材料科学与工程系）；Department of Physics and Applied Physics, Stanford University（斯坦福大学物理与应用物理系）；Geballe Laboratory for Advanced Materials, Stanford University（斯坦福大学Geballe先进材料实验室）；Stanford Institute for Materials and Energy Sciences, Stanford University（斯坦福大学材料与能源科学研究所）；H.H. Wills Physics Laboratory, University of Bristol（布里斯托大学H.H. Wills物理实验室）；Department of Physics, Carnegie Mellon University（卡内基梅隆大学物理系）；Department of Physics and Astrophysics, Howard University（霍华德大学物理与天体物理系）
Hyperspectral images, which record the electromagnetic spectrum for a pixel in the image of a scene, often store hundreds of channels per pixel and contain an order of magnitude more information than a typical similarly-sized color image. Consequently, concomitant with the decreasing cost of capturing these images, there is a need to develop efficient techniques for storing, transmitting, and analyzing hyperspectral images. This paper develops a method for hyperspectral image compression using implicit neural representations where a multilayer perceptron network $Φ_θ$ with sinusoidal activatio…
Sapienza University of Rome（罗马第一大学）；Hochschule Bonn-Rhein-Sieg（波恩-莱茵-锡格应用技术大学）
Representing 3D surfaces as level sets of continuous functions over $\mathbb{R}^3$ is the common denominator of neural implicit representations, which recently enabled remarkable progress in geometric deep learning and computer vision tasks. In order to represent 3D motion within this framework, it is often assumed (either explicitly or implicitly) that the transformations which a surface may undergo are homeomorphic: this is not necessarily true, for instance, in the case of fluid dynamics. In order to represent more general classes of deformations, we propose to apply this theoretical framew…
