Dynamic medical imaging is usually limited in application due to the large radiation doses and longer image scanning and reconstruction times. Existing methods attempt to reduce the dynamic sequence by interpolating the volumes between the acquired image volumes. However, these methods are limited to either 2D images and/or are unable to support large variations in the motion between the image volume sequences. In this paper, we present a spatiotemporal volumetric interpolation network (SVIN) designed for 4D dynamic medical images. SVIN introduces dual networks: first is the spatiotemporal motion network that leverages the 3D convolutional neural network (CNN) for unsupervised parametric volumetric registration to derive spatiotemporal motion field from two-image volumes; the second is the sequential volumetric interpolation network, which uses the derived motion field to interpolate image volumes, together with a new regression-based module to characterize the periodic motion cycles in functional organ structures. We also introduce an adaptive multi-scale architecture to capture the volumetric large anatomy motions. Experimental results demonstrated that our SVIN outperformed state-of-the-art temporal medical interpolation methods and natural video interpolation methods that have been extended to support volumetric images. Our ablation study further exemplified that our motion network was able to better represent the large functional motion compared with the state-of-the-art unsupervised medical registration methods.
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins Medical Institute（约翰斯·霍普金斯医学研究所）
Clear identification of bone structures is crucial for ultrasound-guided lumbar interventions, but it can be challenging due to the complex shapes of the self-shadowing vertebra anatomy and the extensive background speckle noise from the surrounding soft tissue structures. Therefore, we propose to use a patch-like wearable ultrasound solution to capture the reflective bone surfaces from multiple imaging angles and create 3D bone representations for interventional guidance. In this work, we will present our method for estimating the vertebra bone surfaces by using a spatiotemporal U-Net archite…
Schrödinger-Inspired Time-Evolution for 4D Deformation Forecasting
Department of Mathematics, University of Innsbruck（数学系，因斯布鲁克大学）；Department of Radiology, Medical University of Innsbruck（放射学系，因斯布鲁克医学院）；Neuroimaging Research Core Facility, Medical University of Innsbruck（神经影像研究核心设施，因斯布鲁克医学院）
A. K. Choudhury School of Information Technology, University of Calcutta（加尔各答大学A.K.乔杜里信息技术学院）；Indian Statistical Institute（印度统计研究所）
Magnetic Resonance Imaging (MRI) is a technology for non-invasive imaging of anatomical features in detail. It can help in functional analysis of organs of a specimen but it is very costly. In this work, methods for (i) virtual three-dimensional (3D) reconstruction from a single sequence of two-dimensional (2D) slices of MR images of a human spine and brain along a single axis, and (ii) generation of missing inter-slice data are proposed. Our approach helps in preserving the edges, shape, size, as well as the internal tissue structures of the object being captured. The sequence of original 2D …
Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Massachusetts Institute of Technology（麻省理工学院）
Optical coherence tomography (OCT) is a micrometer-scale, volumetric imaging modality that has become a clinical standard in ophthalmology. OCT instruments image by raster-scanning a focused light spot across the retina, acquiring sequential cross-sectional images to generate volumetric data. Patient eye motion during the acquisition poses unique challenges: Non-rigid, discontinuous distortions can occur, leading to gaps in data and distorted topographic measurements. We present a new distortion model and a corresponding fully-automatic, reference-free optimization strategy for computational m…
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；School of Electrical Engineering（电气工程学院）
With the advance in user-friendly and powerful video editing tools, anyone can easily manipulate videos without leaving prominent visual traces. Frame-rate up-conversion (FRUC), a representative temporal-domain operation, increases the motion continuity of videos with a lower frame-rate and is used by malicious counterfeiters in video tampering such as generating fake frame-rate video without improving the quality or mixing temporally spliced videos. FRUC is based on frame interpolation schemes and subtle artifacts that remain in interpolated frames are often difficult to distinguish. Hence, d…
Latent Interpolation Learning Using Diffusion Models for Cardiac Volume Reconstruction
School of Computation, Information and Technology, Technical University Munich（技术大学慕尼黑计算信息学院）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Department of Quantitative Biomedicine, University of Zurich（苏黎世大学定量生物医学系）；Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford（牛津大学生物医学工程研究所）；Imperial College London（伦敦帝国学院）；University of Sheffield（谢菲尔德大学）；NVIDIA（英伟达）；Department of Computing, Imperial College London（伦敦帝国学院计算机系）；Department AIBE of Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔兰根-纽伦堡大学AIBE系）；School of Medicine, Klinikum Rechts der Isar, Technical University of Munich（慕尼黑技术大学医学院，莱纳特医院）
Markerless motion capture algorithms require a 3D body with properly personalized skeleton dimension and/or body shape and appearance to successfully track a person. Unfortunately, many tracking methods consider model personalization a different problem and use manual or semi-automatic model initialization, which greatly reduces applicability. In this paper, we propose a fully automatic algorithm that jointly creates a rigged actor model commonly used for animation - skeleton, volumetric shape, appearance, and optionally a body surface - and estimates the actor's motion from multi-view video i…
Single-Photon Image Super-Resolution (SPISR) aims to recover a high-resolution volumetric photon counting cube from a noisy low-resolution one by computational imaging algorithms. In real-world scenarios, pairs of training samples are often expensive or impossible to obtain. By extending Equivariant Imaging (EI) to volumetric single-photon data, we propose a self-supervised learning framework for the SPISR task. Particularly, using the Poisson unbiased Kullback-Leibler risk estimator and equivariance, our method is able to learn from noisy measurements without ground truths. Comprehensive expe…
Medical University of Vienna（维也纳医科大学）；King's College London（伦敦国王学院）；Nathan Kline Institute（内森·克莱恩研究所）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
Motion correction is an essential preprocessing step in functional Magnetic Resonance Imaging (fMRI) of the fetal brain with the aim to remove artifacts caused by fetal movement and maternal breathing and consequently to suppress erroneous signal correlations. Current motion correction approaches for fetal fMRI choose a single 3D volume from a specific acquisition timepoint with least motion artefacts as reference volume, and perform interpolation for the reconstruction of the motion corrected time series. The results can suffer, if no low-motion frame is available, and if reconstruction does …
Volumetric (3d) images are acquired for many scientific and biomedical purposes using imaging methods such as serial section microscopy, CT scans, and MRI. A frequent step in the analysis and reconstruction of such data is the alignment and registration of images that were acquired in succession along a spatial or temporal dimension. For example, in serial section electron microscopy, individual 2d sections are imaged via electron microscopy and then must be aligned to one another in order to produce a coherent 3d volume. State of the art approaches find image correspondences derived from patc…
DCL-SE: Dynamic Curriculum Learning for Spatiotemporal Encoding of Brain Imaging
University of Chinese Academy of Sciences（中国科学院大学）；Capital Medical University（首都医科大学）；Beijing Tongren Hospital（北京同仁医院）；Beijing Institute of Ophthalmology（北京眼科研究所）；Wannan Medical University（皖南医学院）
Samsung Advanced Institute of Technology (SAIT)（三星先进技术研究院）
Temporal interpolation often plays a crucial role to learn meaningful representations in dynamic scenes. In this paper, we propose a novel method to train spatiotemporal neural radiance fields of dynamic scenes based on temporal interpolation of feature vectors. Two feature interpolation methods are suggested depending on underlying representations, neural networks or grids. In the neural representation, we extract features from space-time inputs via multiple neural network modules and interpolate them based on time frames. The proposed multi-level feature interpolation network effectively cap…
We present Neural Marionette, an unsupervised approach that discovers the skeletal structure from a dynamic sequence and learns to generate diverse motions that are consistent with the observed motion dynamics. Given a video stream of point cloud observation of an articulated body under arbitrary motion, our approach discovers the unknown low-dimensional skeletal relationship that can effectively represent the movement. Then the discovered structure is utilized to encode the motion priors of dynamic sequences in a latent structure, which can be decoded to the relative joint rotations to repres…
GlobalPaint: Spatiotemporal Coherent Video Outpainting with Global Feature Guidance
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（人机混合增强智能国家重点实验室，人工智能与机器人研究院，西安交通大学）；Microsoft Research Asia（微软亚洲研究院）；University of Science and Technology of China（中国科学技术大学）
University of Vienna（维也纳大学）；Austrian Academy of Sciences（奥地利科学院）
We extend the concept of optical flow with spatiotemporal regularisation to a dynamic non-Euclidean setting. Optical flow is traditionally computed from a sequence of flat images. The purpose of this paper is to introduce variational motion estimation for images that are defined on an evolving surface. Volumetric microscopy images depicting a live zebrafish embryo serve as both biological motivation and test data.
What, Where, and How: Probing Spatiotemporal Representations in Video Foundation Models
Lassonde School of Engineering（拉桑德工程学院）；York University（约克大学）；Vector Institute（矢量研究院）；FAIR, Meta Superintelligence Labs（Meta超级智能实验室FAIR）；Mila（米拉研究院）；McGill University（麦吉尔大学）；Goodfire AI（Goodfire人工智能公司）
An Anisotropic Cross-View Texture Transfer with Multi-Reference Non-Local Attention for CT Slice Interpolation
Department of Artificial Intelligence, Gachon University（人工智能系，加荣大学）；Department of Electrical Engineering, Korea University（电气工程系，韩国大学）；Department of Urology, The Catholic University of Korea（泌尿科，韩国天主大学）
MiVID: Multi-Strategic Self-Supervision for Video Frame Interpolation using Diffusion Model
School of Computer Engineering, KIIT Deemed to be University（计算机工程学院，KIIT被认定大学）；Department of Computer Science and Engineering, National Institute of Technology（计算机科学与工程系，国家理工学院）
TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion
National Key Lab of Multispectral Information Intelligent Processing Technology（multispectral information intelligent processing technology key laboratory）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（人工智能与自动化学院，华中科技大学）；School of Electrical and Electronic Engineering, Nanyang Technological University（电子与电气工程学院，南洋理工大学）
Acquiring High Resolution (HR) Magnetic Resonance (MR) images requires the patient to remain still for long periods of time, which causes patient discomfort and increases the probability of motion induced image artifacts. A possible solution is to acquire low resolution (LR) images and to process them with the Super Resolution Generative Adversarial Network (SRGAN) to create a super-resolved version. This work applies SRGAN to MR images of the prostate and performs three experiments. The first experiment explores improving the in-plane MR image resolution by factors of 4 and 8, and shows that,…
Towards Voxel Spacing Consistency for Medical Image Segmentation
Institute of Medical Robotics, Shanghai Jiao Tong University（上海交通大学医疗机器人研究院）；Department of Ultrasound, Shanghai Chest Hospital（上海市胸科医院超声科）；CAMPAR, Technical University of Munich（慕尼黑工业大学CAMPAR）
Universitat Politècnica de Catalunya - BarcelonaTech（加泰罗尼亚理工大学 - 巴塞罗那理工大学）
In this work we propose an adversarial learning approach to generate high resolution MRI scans from low resolution images. The architecture, based on the SRGAN model, adopts 3D convolutions to exploit volumetric information. For the discriminator, the adversarial loss uses least squares in order to stabilize the training. For the generator, the loss function is a combination of a least squares adversarial loss and a content term based on mean square error and image gradients in order to improve the quality of the generated images. We explore different solutions for the upsampling phase. We pre…
Continuous sPatial-Temporal Deformable Image Registration (CPT-DIR) for motion modelling in radiotherapy: beyond classic voxel-based methods
Center for Proton Therapy, Paul Scherrer Institut（保罗·施特格研究所质子治疗中心）；Department of Computer Science, ETH Zürich（苏黎世联邦理工学院计算机科学系）；Department of Physics, ETH Zürich（苏黎世联邦理工学院物理系）；Department of Automation, Tsinghua Univeristy（清华大学自动化系）；S-Lab, Nanyang Technological University（南洋理工大学S实验室）
