High annotation costs and limited labels for dense 3D medical imaging tasks have recently motivated an assortment of 3D self-supervised pretraining methods that improve transfer learning performance. However, these methods commonly lack spatial awareness despite its centrality in enabling effective 3D image analysis. More specifically, position, scale, and orientation are not only informative but also automatically available when generating image crops for training. Yet, to date, no work has proposed a pretext task that distills all key spatial features. To fulfill this need, we develop a new self-supervised method, VectorPOSE, which promotes better spatial understanding with two novel pretext tasks: Vector Prediction (VP) and Boundary-Focused Reconstruction (BFR). VP focuses on global spatial concepts (i.e., properties of 3D patches) while BFR addresses weaknesses of recent reconstruction methods to learn more effective local representations. We evaluate VectorPOSE on three 3D medical image segmentation tasks, showing that it often outperforms state-of-the-art methods, especially in limited annotation settings.
Existing techniques to encode spatial invariance within deep convolutional neural networks (CNNs) apply the same warping field to all the feature channels. This does not account for the fact that the individual feature channels can represent different semantic parts, which can undergo different spatial transformations w.r.t. a canonical configuration. To overcome this limitation, we introduce a learnable module, the volumetric transformer network (VTN), that predicts channel-wise warping fields so as to reconfigure intermediate CNN features spatially and channel-wisely. We design our VTN as an…
High-quality 3D object recognition is an important component of many vision and robotics systems. We tackle the object recognition problem using two data representations, to achieve leading results on the Princeton ModelNet challenge. The two representations: 1. Volumetric representation: the 3D object is discretized spatially as binary voxels - $1$ if the voxel is occupied and $0$ otherwise. 2. Pixel representation: the 3D object is represented as a set of projected 2D pixel images. Current leading submissions to the ModelNet Challenge use Convolutional Neural Networks (CNNs) on pixel represe…
Raptor: Scalable Train-Free Embeddings for 3D Medical Volumes Leveraging Pretrained 2D Foundation Models
Computer Science Department, University of California, Los Angeles（计算机科学系，加州大学洛杉矶分校）；Department of Computational Medicine, David Geffen School of Medicine, University of California, Los Angeles（计算医学系，大卫·格芬医学院，加州大学洛杉矶分校）；Bioinformatics Interdepartmental Program, University of California, Los Angeles（生物信息学跨学科项目，加州大学洛杉矶分校）；Department of Human Genetics, University of California, Los Angeles（人类遗传学系，加州大学洛杉矶分校）
BiSegMamba: Efficient Bidirectional Tri-Oriented Mamba for 3D Medical Image Segmentation
School of Computer Science and Engineering, Beihang University（北航计算机科学与工程学院）；State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北航虚拟现实技术与系统国家重点实验室）
Real-time rendering and animation of humans is a core function in games, movies, and telepresence applications. Existing methods have a number of drawbacks we aim to address with our work. Triangle meshes have difficulty modeling thin structures like hair, volumetric representations like Neural Volumes are too low-resolution given a reasonable memory budget, and high-resolution implicit representations like Neural Radiance Fields are too slow for use in real-time applications. We present Mixture of Volumetric Primitives (MVP), a representation for rendering dynamic 3D content that combines the…
Stony Brook University, State University of New York（纽约州立大学石溪分校）；University College Dublin（都柏林大学学院）
We report a light-field based method that allows the optical encryption of three-dimensional (3D) volumetric information at the microscopic scale in a single 2D light-field image. The system consists of a microlens array and an array of random phase/amplitude masks. The method utilizes a wave optics model to account for the dominant diffraction effect at this new scale, and the system point-spread function (PSF) serves as the key for encryption and decryption. We successfully developed and demonstrated a deconvolution algorithm to retrieve spatially multiplexed discrete and continuous volumetr…
Universitat Pompeu Fabra（庞培法布拉大学）；Normandie Univ（诺曼底大学）；UNICAEN（卡昂大学）；CHU de Caen Normandie（卡昂诺曼底大学医院）；Inserm Comete（法国国家健康与医学研究院Comete实验室）；GIP Cyceron（Cyceron集团利益公共机构）；Philips Research（飞利浦研究院）；ICREA（加泰罗尼亚高等研究院）；IDIBAPS（巴塞罗那生物医学研究所）；KULeuven（鲁汶大学）
3D echocardiography is an increasingly popular tool for assessing cardiac remodelling in the right ventricle (RV). It allows quantification of the cardiac chambers without any geometric assumptions, which is the main weakness of 2D echocardiography. However, regional quantification of geometry and function is limited by the lower spatial and temporal resolution and the scarcity of identifiable anatomical landmarks. We developed a technique for regionally assessing the 3 relevant RV regions: apical, inlet and outflow. The method's inputs are end-diastolic (ED) and end-systolic (ES) segmented 3D…
NVIDIA（英伟达）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
In this work, we address the problem of jointly estimating albedo, normals, depth and 3D spatially-varying lighting from a single image. Most existing methods formulate the task as image-to-image translation, ignoring the 3D properties of the scene. However, indoor scenes contain complex 3D light transport where a 2D representation is insufficient. In this paper, we propose a unified, learning-based inverse rendering framework that formulates 3D spatially-varying lighting. Inspired by classic volume rendering techniques, we propose a novel Volumetric Spherical Gaussian representation for light…
We present a lightweight solution to recover 3D pose from multi-view images captured with spatially calibrated cameras. Building upon recent advances in interpretable representation learning, we exploit 3D geometry to fuse input images into a unified latent representation of pose, which is disentangled from camera view-points. This allows us to reason effectively about 3D pose across different views without using compute-intensive volumetric grids. Our architecture then conditions the learned representation on camera projection operators to produce accurate per-view 2d detections, that can be …
SAND: Spatially Adaptive Network Depth for Fast Sampling of Neural Implicit Surfaces
Shandong University China；City University of Hong Kong China；Hong Kong University of Science；The University of Hong Kong China；Texas A\&M University USA；Shandong University；City University of Hong Kong；The University of Hong Kong；Texas A\&M University
Institute of Computer Science, University of Tartu（塔尔图大学计算机科学研究所）；Wellcome Sanger Institute（惠康桑格研究所）；PerkinElmer Cellular Technologies（珀金埃尔默细胞技术公司）
Biomedical images are noisy. The imaging equipment itself has physical limitations, and the consequent experimental trade-offs between signal-to-noise ratio, acquisition speed, and imaging depth exacerbate the problem. Denoising is, therefore, an essential part of any image processing pipeline, and convolutional neural networks are currently the method of choice for this task. One popular approach, Noise2Noise, does not require clean ground truth, and instead, uses a second noisy copy as a training target. Self-supervised methods, like Noise2Self and Noise2Void, relax data requirements by lear…
Human motion retargeting aims to transfer the motion of one person in a "driving" video or set of images to another person. Existing efforts leverage a long training video from each target person to train a subject-specific motion transfer model. However, the scalability of such methods is limited, as each model can only generate videos for the given target subject, and such training videos are labor-intensive to acquire and process. Few-shot motion transfer techniques, which only require one or a few images from a target, have recently drawn considerable attention. Methods addressing this tas…
We present a deep learning solution for estimating the incident illumination at any 3D location within a scene from an input narrow-baseline stereo image pair. Previous approaches for predicting global illumination from images either predict just a single illumination for the entire scene, or separately estimate the illumination at each 3D location without enforcing that the predictions are consistent with the same 3D scene. Instead, we propose a deep learning model that estimates a 3D volumetric RGBA model of a scene, including content outside the observed field of view, and then uses standar…
College of Mathematics and Computer Science Fuzhou University（福州大学数学与计算机学院）；Imperial Vision（帝国视觉）
Detection and segmentation of the hippocampal structures in volumetric brain images is a challenging problem in the area of medical imaging. In this paper, we propose a two-stage 3D fully convolutional neural network that efficiently detects and segments the hippocampal structures. In particular, our approach first localizes the hippocampus from the whole volumetric image while obtaining a proposal for a rough segmentation. After localization, we apply the proposal as an enhancement mask to extract the fine structure of the hippocampus. The proposed method has been evaluated on a public datase…
Tel-Aviv University（特拉维夫大学）；Sheba Medical Center（舍巴医疗中心）
Pulmonary embolus (PE) refers to obstruction of pulmonary arteries by blood clots. PE accounts for approximately 100,000 deaths per year in the United States alone. The clinical presentation of PE is often nonspecific, making the diagnosis challenging. Thus, rapid and accurate risk stratification is of paramount importance. High-risk PE is caused by right ventricular (RV) dysfunction from acute pressure overload, which in return can help identify which patients require more aggressive therapy. Reconstructed four-chamber views of the heart on chest CT can detect right ventricular enlargement. C…
MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation
Department of Biomedical Engineering, University of Florida（佛罗里达大学生物医学工程系）；Department of Radiology, University of Florida（佛罗里达大学放射学系）；Research Computing, University of Florida（佛罗里达大学研究计算中心）；Department of Medicine, University of Florida（佛罗里达大学医学系）；Department of Radiology, UC San Francisco（旧金山大学放射学系）
Volumetric deep learning approach towards stereo matching aggregates a cost volume computed from input left and right images using 3D convolutions. Recent works showed that utilization of extracted image features and a spatially varying cost volume aggregation complements 3D convolutions. However, existing methods with spatially varying operations are complex, cost considerable computation time, and cause memory consumption to increase. In this work, we construct Guided Cost volume Excitation (GCE) and show that simple channel excitation of cost volume guided by image can improve performance c…
Retrieval-Augmented Anatomical Guidance for Text-to-CT Generation
Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Rome, Italy（人工智能与计算机系统单位，工程系，罗马生物医学大学，意大利罗马）；Department of Diagnostics and Intervention, Biomedical Engineering and Radiation Physics, Umeå University, Umeå, Sweden（诊断与介入系，生物医学工程与放射物理，乌梅大学，瑞典乌梅）
