提出 DiffPoint 架构，结合 ViT 与扩散模型，通过将含噪点云划分为不规则块并作为 token 处理，在单视图和多视图点云重建任务上取得最先进结果，并引入灵活特征融合模块，验证了统一架构改进 3D 重建的可行性。
摘要：
随着 2D 到 3D 重建任务在各种现实场景中获得重要关注，生成高质量点云变得至关重要。尽管深度学习模型近期在生成点云方面取得了成功，但由于图像与点云之间的差异，生成高保真结果仍面临挑战。虽然视觉 Transformer（ViT）和扩散模型在各种视觉任务中展现出潜力，但它们在从图像重建点云方面的优势尚未得到证明。在本文中，我们首先提出了一种简洁且强大的架构 DiffPoint，将 ViT 和扩散模型结合用于点云重建任务。在每个扩散步骤中，我们将含噪点云划分为不规则块。然后，使用将所有输入视为 token（包括时间信息、图像嵌入和含噪块）的标准 ViT 主干网络，我们训练模型基于输入图像预测目标点。我们在单视图和多视图重建任务上评估了 DiffPoint，并取得了最先进的结果。此外，我们引入了一个统一且灵活的特征融合模块，用于聚合来自单个或多个输入图像的图像特征。进一步地，我们的工作证明了跨语言和图像应用统一架构以改进 3D 重建任务的可行性。
英文摘要：
As the task of 2D-to-3D reconstruction has gained significant attention in various real-world scenarios, it becomes crucial to be able to generate high-quality point clouds. Despite the recent success of deep learning models in generating point clouds, there are still challenges in producing high-fidelity results due to the disparities between images and point clouds. While vision transformers (ViT) and diffusion models have shown promise in various vision tasks, their benefits for reconstructing point clouds from images have not been demonstrated yet. In this paper, we first propose a neat and powerful architecture called DiffPoint that combines ViT and diffusion models for the task of point cloud reconstruction. At each diffusion step, we divide the noisy point clouds into irregular patches. Then, using a standard ViT backbone that treats all inputs as tokens (including time information, image embeddings, and noisy patches), we train our model to predict target points based on input images. We evaluate DiffPoint on both single-view and multi-view reconstruction tasks and achieve state-of-the-art results. Additionally, we introduce a unified and flexible feature fusion module for aggregating image features from single or multiple input images. Furthermore, our work demonstrates the feasibility of applying unified architectures across languages and images to improve 3D reconstruction tasks.
Temporal-Spatial Tubelet Embedding for Cloud-Robust MSI Reconstruction using MSI-SAR Fusion: A Multi-Head Self-Attention Video Vision Transformer Approach
ViTok-v2: Scaling Native Resolution Auto-Encoders to 5 Billion Parameters
University of Texas, Austin（德克萨斯大学奥斯汀分校）；University of Washington（华盛顿大学）；Stanford University（斯坦福大学）；Spellbrush；Meta Superintelligence Labs（Meta超智能实验室）；Georgia Institute of Technology（佐治亚理工学院）
Multi-Angle Point Cloud-VAE: Unsupervised Feature Learning for 3D Point Clouds from Multiple Angles by Joint Self-Reconstruction and Half-to-Half Prediction
Tsinghua University（清华大学）；University of Maryland, College Park（马里兰大学帕克分校）；Beijing National Research Center for Information Science and Technology (BNRist)（北京信息科学与技术国家研究中心（BNRist））
Indoor 3D Reconstruction with an Unknown Camera-Projector Pair
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology（空天地海一体化大数据应用技术国家工程实验室）；College of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）
Masked Clustering Prediction for Unsupervised Point Cloud Pre-training
University of Pisa（比萨大学）；University of Trento（特伦托大学）；Shanghai Jiao Tong University（上海交通大学）；Peking University（北京大学）；Fondazione Bruno Kessler（布鲁诺·凯索尔基金会）
ViT-NeBLa: A Hybrid Vision Transformer and Neural Beer-Lambert Framework for Single-View 3D Reconstruction of Oral Anatomy from Panoramic Radiographs
Artificial Intelligence and Image Processing Lab., Department of Information and Communication Engineering, Sun Moon University, South Korea（人工智能与图像处理实验室，信息与通信工程系，顺 Moon 大学，韩国）；Department of Physics and Engineering Physics, Tulane University, New Orleans, LA 70118, USA（物理系与工程物理系，路易斯安那大学，新奥尔良，路易斯安那州 70118，美国）
Boosting ViT-based MRI Reconstruction from the Perspectives of Frequency Modulation, Spatial Purification, and Scale Diversification
Fudan University（复旦大学）；Digital Medical Research Center（数字医学研究中心）；School of Basic Medical Science（基础医学院）；Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention（上海医学影像计算与计算机辅助干预重点实验室）；Academy for Engineering and Technology（工程与技术学院）
EndoWave: 4D Gaussian Splatting with Rational Wavelet for Endoscopic Reconstruction
Xi'an Jiaotong-Liverpool University（西安交通大学利物浦大学）；University of Liverpool（利物浦大学）；The Chinese University of Hong Kong（香港中文大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Zhejiang University of Technology（浙江工业大学）
TryOffDiff: Virtual-Try-Off via High-Fidelity Garment Reconstruction using Diffusion Models
Machine Learning Group, CITEC Bielefeld University Germany（比勒菲尔德大学机器学习小组、CITEC）；Institute of Mathematics Technical University Berlin Germany（柏林技术大学数学研究所）
LoFi: Neural Local Fields for Scalable Image Reconstruction
University of Basel（巴塞尔大学）；Mullard Space Science Laboratory (MSSL), University College London (UCL)（伦敦大学学院穆拉德空间科学实验室）；IRFU, CEA, Université Paris-Saclay（法国原子能和替代能源委员会红外与光子学研究所，巴黎萨克雷大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
ViSIR: Vision Transformer Single Image Reconstruction Method for Earth System Models
Computer Science, Texas State University, San Marcos, 78666, Texas（计算机科学，德克萨斯州立大学，圣马科斯，德克萨斯州）；Chemical Engineering, University of Utah, Salt Lake City, 84112, Utah（化学工程，犹他大学，盐湖城，犹他州）
Magnetic Resonance Image Processing Transformer for General Accelerated Image Reconstruction
Boston University（波士顿大学）；Photonics Center（光子学中心）；Chobanian & Avedisian School of Medicine（乔巴尼亚与阿韦迪西安医学院）；Boston University Medical Campus（波士顿大学医学校区）；Rafik B. Hariri Institute for Computing and Computational Science & Engineering（拉菲克·B·哈里里计算与计算科学与工程研究所）
Hi-End-MAE: Hierarchical encoder-driven masked autoencoders are stronger vision learners for medical image segmentation
University of Science and Technology of China (USTC)（中国科学技术大学）；Suzhou Institute for Advanced Research, USTC（中国科学技术大学苏州高等研究院）；Stanford University（斯坦福大学）；State Key Laboratory of Precision and Intelligent Chemistry, USTC（中国科学技术大学精准与智能化学国家重点实验室）；Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS（中国科学院计算技术研究所中国科学院智能信息处理重点实验室）
Enhancing Global Sensitivity and Uncertainty Quantification in Medical Image Reconstruction with Monte Carlo Arbitrary-Masked Mamba
Imperial College London（帝国理工学院）；Royal Brompton Hospital（皇家布朗普顿医院）；College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）；School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区生物医学工程学院）；Human Phenome Institute, Fudan University（复旦大学人类表型组研究院）；Inselspital, University of Bern（伯尔尼大学因塞尔医院）；Technical University of Munich（慕尼黑工业大学）
LifWavNet: Lifting Wavelet-based Network for Non-contact ECG Reconstruction from Radar
Rekhi Centre of Excellence for the Science of Happiness, IIT Kharagpur, India（幸福科学卓越中心，IIT哈里科普，印度）；Department of Electrical Engineering, IIT Kharagpur, India（电气工程系，IIT哈里科普，印度）；Department of Electronics and Electrical Communication Engineering, IIT Kharagpur, India（电子与电子通信工程系，IIT哈里科普，印度）
Dual-domain Multi-path Self-supervised Diffusion Model for Accelerated MRI Reconstruction
Department of Radiology, Northwestern University（放射科，西北大学）；Department of Biomedical Engineering, Huazhong University of Science and Technology（生物医学工程系，华中科技大学）
UPCMR: A Universal Prompt-guided Model for Random Sampling Cardiac MRI Reconstruction
Leiden University Medical Center（莱顿大学医学中心）；Cardiologs；Philips（飞利浦）；Eindhoven University of Technology（埃因霍温理工大学）；Philips Innovative Technologies（飞利浦创新技术）
Delving into Masked Autoencoders for Multi-Label Thorax Disease Classification
Johns Hopkins University（约翰斯·霍普金斯大学）
视觉Transformer（ViT）由于其在许多视觉任务中的巨大可扩展性、计算效率和令人信服的性能而成为最流行的神经架构之一。然而，ViT在医疗任务上表现出不如卷积神经网络（CNN）的性能，这是由于其数据饥渴的性质和缺乏注释的医疗数据。在本文中，我们使用掩码自编码器（MAE）对266，340张胸部X光片进行了预训练ViT，MAE从每张图像的一小部分中重建丢失的像素。为了进行比较，CNN也使用先进的自监督方法（例如，MoCo v2）。结果表明，我们的预训练ViT在多标签胸部疾病分类方面表现优于（有时更好）最先进的CNN（DenseNet-121）。这一表现归功于从我们的经验研究中提取的用于预训练和微调ViT的强大配方。预训练配方表明，与自然成像相比，医学重建需要更小比例的图像（10% vs. 25%）和更适度的随机调整大小的裁剪范围（0.5~1.0 vs. 0.2~1.0）。此外，我们注意到，只要可能，域内迁移学习是优选的。微调配方公开了逐层LR衰减、RandAug幅度和DropPath速率是要考虑的重要因素。我们希望这项研究可以指导未来的研究应用Transformers到更大的各种医学成像任务。
