Enhancing Free-hand 3D Photoacoustic and Ultrasound Reconstruction using Deep Learning
Pusan National University（釜山国立大学）；Busan Medical Center（釜山医疗中心）
SiYeoul Lee, SeonHo Kim, Minkyung Seo, SeongKyu Park, Salehin Imrus, Kambaluru Ashok, DongEon Lee, Chunsu Park, SeonYeong Lee, Jiye Kim, Jae-Heung Yoo, MinWoo Kim
This study introduces a motion-based learning network with a global-local self-attention module (MoGLo-Net) to enhance 3D reconstruction in handheld photoacoustic and ultrasound (PAUS) imaging. Standard PAUS imaging is often limited by a narrow field of view and the inability to effectively visualize complex 3D structures. The 3D freehand technique, which aligns sequential 2D images for 3D reconstruction, faces significant challenges in accurate motion estimation without relying on external positional sensors. MoGLo-Net addresses these limitations through an innovative adaptation of the self-attention mechanism, which effectively exploits the critical regions, such as fully-developed speckle area or high-echogenic tissue area within successive ultrasound images to accurately estimate motion parameters. This facilitates the extraction of intricate features from individual frames. Additionally, we designed a patch-wise correlation operation to generate a correlation volume that is highly correlated with the scanning motion. A custom loss function was also developed to ensure robust learning with minimized bias, leveraging the characteristics of the motion parameters. Experimental evaluations demonstrated that MoGLo-Net surpasses current state-of-the-art methods in both quantitative and qualitative performance metrics. Furthermore, we expanded the application of 3D reconstruction technology beyond simple B-mode ultrasound volumes to incorporate Doppler ultrasound and photoacoustic imaging, enabling 3D visualization of vasculature. The source code for this study is publicly available at: https://github.com/guhong3648/US3D
It has previously been demonstrated that model-based reconstruction methods relying on a priori knowledge of the imaging point spread function (PSF) coupled to sparsity priors on the object to image can provide super-resolution in photoacoustic (PA) or in ultrasound (US) imaging. Here, we experimentally show that such reconstruction also leads to super-resolution in both PA and US imaging with arrays having much less elements than used conventionally (sparse arrays). As a proof of concept, we obtained super-resolution PA and US cross-sectional images of microfluidic channels with only 8 elemen…
Rensselaer Polytechnic Institute（伦斯勒理工学院）；Michigan State University（密歇根州立大学）
Photoacoustic tomography seeks to reconstruct an acoustic initial pressure distribution from the measurement of the ultrasound waveforms. Conventional methods assume a-prior knowledge of the sound speed distribution, which practically is unknown. One way to circumvent the issue is to simultaneously reconstruct both the acoustic initial pressure and speed. In this article, we develop a novel data-driven method that integrates an advanced deep neural network through model-based iteration. The image of the initial pressure is significantly improved in our numerical simulation.
Photoacoustic tomography (PAT) is a newly developed medical imaging modality, which combines the advantages of pure optical imaging and ultrasound imaging, owning both high optical contrast and deep penetration depth. Very recently, PAT is studied in human brain imaging. Nevertheless, while ultrasound waves are passing through the human skull tissues, the strong acoustic attenuation and aberration will happen, which causes photoacoustic signals' distortion. In this work, we use 10 magnetic resonance angiography (MRA) human brain volumes, and manually segment them to obtain the 2D human brain n…
ShanghaiTech University（上海科技大学）；University of Chinese Academy of Sciences（中国科学院大学）
Photoacoustic imaging (PAI) is a non-invasive imaging modality that detects the ultrasound signal generated from tissue with light excitation. Photoacoustic computed tomography (PACT) uses unfocused large-area light to illuminate the target with ultrasound transducer array for PA signal detection. Limited-view issue could cause a low-quality image in PACT due to the limitation of geometric condition. The model-based method is used to resolve this problem, which contains different regularization. To adapt fast and high-quality reconstruction of limited-view PA data, in this paper, a model-based…
Photoacoustic (PA) endoscopy has shown significant potential for clinical diagnosis and surgical guidance. Multimode fibres (MMFs) are becoming increasing attractive for the development of miniature endoscopy probes owing to ultrathin size, low cost and diffraction-limited spatial resolution enabled by wavefront shaping. However, current MMF-based PA endomicroscopy probes are either limited by a bulky ultrasound detector or a low imaging speed which hindered their usability. In this work, we report the development of a highly miniaturised and high-speed PA endomicroscopy probe that is integrat…
German Cancer Research Center (DKFZ)（德国癌症研究中心（DKFZ））；Heidelberg University（海德堡大学）；HIDSS4Health - Helmholtz Information and Data Science School for Health（HIDSS4Health——亥姆霍兹健康信息与数据科学学院）；Cancer Research UK Cambridge Institute（英国癌症研究院剑桥研究所）；University of Cambridge（剑桥大学）
Photoacoustic (PA) imaging has the potential to revolutionize functional medical imaging in healthcare due to the valuable information on tissue physiology contained in multispectral photoacoustic measurements. Clinical translation of the technology requires conversion of the high-dimensional acquired data into clinically relevant and interpretable information. In this work, we present a deep learning-based approach to semantic segmentation of multispectral photoacoustic images to facilitate image interpretability. Manually annotated photoacoustic {and ultrasound} imaging data are used as refe…
Conventional photoacoustic imaging may suffer from the limited view and bandwidth of ultrasound transducers. A deep learning approach is proposed to handle these problems and is demonstrated both in simulations and in experiments on a multi-scale model of leaf skeleton. We employed an experimental approach to build the training and the test sets using photographs of the samples as ground truth images. Reconstructions produced by the neural network show a greatly improved image quality as compared to conventional approaches. In addition, this work aimed at quantifying the reliability of the neu…
University of Michigan-Shanghai Jiao Tong University Joint Institute, Shanghai Jiao Tong University（上海交通大学密西根学院）；Biomedical Engineering Department, Peking University（北京大学生物医学工程系）
As a hybrid imaging technology, photoacoustic microscopy (PAM) imaging suffers from noise due to the maximum permissible exposure of laser intensity, attenuation of ultrasound in the tissue, and the inherent noise of the transducer. De-noising is a post-processing method to reduce noise, and PAM image quality can be recovered. However, previous de-noising techniques usually heavily rely on mathematical priors as well as manually selected parameters, resulting in unsatisfactory and slow de-noising performance for different noisy images, which greatly hinders practical and clinical applications.…
Photoacoustic imaging has shown great potential for guiding minimally invasive procedures by accurate identification of critical tissue targets and invasive medical devices (such as metallic needles). The use of light emitting diodes (LEDs) as the excitation light sources accelerates its clinical translation owing to its high affordability and portability. However, needle visibility in LED-based photoacoustic imaging is compromised primarily due to its low optical fluence. In this work, we propose a deep learning framework based on U-Net to improve the visibility of clinical metallic needles w…
Neuromuscular diseases (NMDs) cause a significant burden for both healthcare systems and society. They can lead to severe progressive muscle weakness, muscle degeneration, contracture, deformity and progressive disability. The NMDs evaluated in this study often manifest in early childhood. As subtypes of disease, e.g. Duchenne Muscular Dystropy (DMD) and Spinal Muscular Atrophy (SMA), are difficult to differentiate at the beginning and worsen quickly, fast and reliable differential diagnosis is crucial. Photoacoustic and ultrasound imaging has shown great potential to visualize and quantify th…
Recently we demonstrated an integrated photoacoustic (PA) and ultrasound (PAUS) system using a kHz-rate wavelength-tunable laser and a swept-beam delivery approach. It irradiates a medium using a narrow laser beam sweeping at high repetition rate over the desired imaging area, in contrast to the conventional PA approach using broad beam illumination at a low repetition. One significant advantage of this approach is that the fundamental problem of decoupling local light absorption at a point from optical fluence at the same point can be solved. Here, we present a fluence compensation method and…
Photoacoustic computed tomography (PACT) combines the optical contrast of optical imaging and the penetrability of sonography. In this work, we develop a novel PACT system to provide real-time imaging, which is achieved by a 120-elements ultrasound array only using a single data acquisition (DAQ) channel. To reduce the channel number of DAQ, we superimpose 30 nearby channels' signals together in the analog domain, and shrinking to 4 channels of data (120/30=4). Furthermore, a four-to-one delay-line module is designed to combine these four channels' data into one channel before entering the sin…
ShanghaiTech University（上海科技大学）；Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）
Photoacoustic imaging (PAI) is an emerging non-invasive imaging modality combining the advantages of deep ultrasound penetration and high optical contrast. Image reconstruction is an essential topic in PAI, which is unfortunately an ill-posed problem due to the complex and unknown optical/acoustic parameters in tissue. Conventional algorithms used in PAI (e.g., delay-and-sum) provide a fast solution while many artifacts remain, especially for linear array probe with limited-view issue. Convolutional neural network (CNN) has shown state-of-the-art results in computer vision, and more and more w…
Based on photoacoustic effect, photoacoustic tomography is developing very fast in recent years, and becoming an important imaging tool for both preclinical and clinical studies. With enough ultrasound transducers placed around the biological tissue, PAT can provide both deep penetration and high image contrast by hybrid usage of light and sound. However, considering space and measurement environmental limitations, transducers are always placed in a limited-angle way, which means that the other side without transducer coverage suffers severe information loss. With conventional image reconstruc…
Ultrasound simulation based on ray tracing enables the synthesis of highly realistic images. It can provide an interactive environment for training sonographers as an educational tool. However, due to high computational demand, there is a trade-off between image quality and interactivity, potentially leading to sub-optimal results at interactive rates. In this work we introduce a deep learning approach based on adversarial training that mitigates this trade-off by improving the quality of simulated images with constant computation time. An image-to-image translation framework is utilized to tr…
Combined fluorescence and photoacoustic imaging of tozuleristide in muscle tissue in vitro -- toward optically-guided solid tumor surgery: feasibility studies
本研究提出联合NIRF与快速扫描PAUS成像方法，在体外牛肌肉组织中实现了深达34 mm的20 uM tozuleristide识别，有效克服NIRF深度限制，为光学引导实体瘤手术的综合术前与术中检测及宽边缘切除提供了可行方案。
Acoustic-resolution photoacoustic microscopy (AR-PAM) is a promising tool for microvascular imaging. In the focal region, resolution of AR-PAM is determined by the ultrasound transducer and ultimately limited by acoustic diffraction. In the out-of-focus region, resolution deteriorates with increasing distance from the focal plane, which restricts depth of focus (DOF). Besides, a trade-off exists between resolution and DOF. Previously, synthetic aperture focusing technique (SAFT) and/or deconvolution methods have been demonstrated to enhance AR-PAM images. However, they suffer from issues in lo…
Coordinate-based Speed of Sound Recovery for Aberration-Corrected Photoacoustic Computed Tomography
Northwestern University（西北大学）；California Institute of Technology（加州理工学院）；Tsinghua University（清华大学）；NSF-Simons AI Institute for the Sky (SkAI)（NSF-模拟斯通AI天空研究所（SkAI））
German Cancer Research Center (DKFZ)（德国癌症研究中心（DKFZ））；Heidelberg University（海德堡大学）；Heidelberg University Hospital（海德堡大学医院）
Purpose: Photoacoustic tomography (PAT) is a novel imaging technique that can spatially resolve both morphological and functional tissue properties, such as the vessel topology and tissue oxygenation. While this capacity makes PAT a promising modality for the diagnosis, treatment and follow-up of various diseases, a current drawback is the limited field-of-view (FoV) provided by the conventionally applied 2D probes. Methods: In this paper, we present a novel approach to 3D reconstruction of PAT data (Tattoo tomography) that does not require an external tracking system and can smoothly be integ…
Shanghai Jiao Tong University（上海交通大学）；Shanghai General Hospital（上海市第一人民医院）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）；AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院）
Much progress has been made in the deep neural network (DNN) based diagnosis of mass lesions breast ultrasound (BUS) images. However, the non-mass lesion is less investigated because of the limited data. Based on the insight that mass data is sufficient and shares the same knowledge structure with non-mass data of identifying the malignancy of a lesion based on the ultrasound image, we propose a novel transfer learning framework to enhance the generalizability of the DNN model for non-mass BUS with the help of mass BUS. Specifically, we train a shared DNN with combined non-mass and mass data. …
UltraDfeGAN: Detail-Enhancing Generative Adversarial Networks for High-Fidelity Functional Ultrasound Synthesis
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Department of Information and Communication, Institute of Technology of Cambodia（柬埔寨技术学院信息与通信系）；Department of Telecommunication and Network Engineering, Institute of Technology of Cambodia（柬埔寨技术学院电信与网络工程系）
Semantic Scene Graph for Ultrasound Image Explanation and Scanning Guidance
Computer Aided Medical Procedures (CAMP)（计算机辅助医疗程序）；TU Munich, Germany（慕尼黑工业大学，德国）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；The Chinese University of Hong Kong, Hong Kong, China（香港中文大学，中国香港）
Temporal Representation Learning for Real-Time Ultrasound Analysis
Department of Computer Science, ETH Zurich, Switzerland（苏黎世联邦理工学院计算机科学系）；Department of Biomedical Engineering, University of Basel, Switzerland（巴塞尔大学生物医学工程系）
Hybrid Attention Network for Accurate Breast Tumor Segmentation in Ultrasound Images
School of Information Engineering, Xi’an Eurasia University（信息工程学院，西安欧亚大学）；Department of Computer Science, University of Engineering and Technology Lahore（计算机科学系，拉合尔工程科技大学）
