Supervised learning with deep convolutional neural networks (DCNNs) has seen huge adoption in stereo matching. However, the acquisition of large-scale datasets with well-labeled ground truth is cumbersome and labor-intensive, making supervised learning-based approaches often hard to implement in practice. To overcome this drawback, we propose a robust and effective self-supervised stereo matching approach, consisting of a pyramid voting module (PVM) and a novel DCNN architecture, referred to as OptStereo. Specifically, our OptStereo first builds multi-scale cost volumes, and then adopts a recurrent unit to iteratively update disparity estimations at high resolution; while our PVM can generate reliable semi-dense disparity images, which can be employed to supervise OptStereo training. Furthermore, we publish the HKUST-Drive dataset, a large-scale synthetic stereo dataset, collected under different illumination and weather conditions for research purposes. Extensive experimental results demonstrate the effectiveness and efficiency of our self-supervised stereo matching approach on the KITTI Stereo benchmarks and our HKUST-Drive dataset. PVStereo, our best-performing implementation, greatly outperforms all other state-of-the-art self-supervised stereo matching approaches. Our project page is available at sites.google.com/view/pvstereo.
Pyramid Multi-view Stereo Net with Self-adaptive View Aggregation
PKU（北京大学）；HKU（香港大学）；Tencent（腾讯）；Kwai Inc.（快手）
本文提出了一种高效的金字塔多视图立体视觉（MVS）网络，该网络具有自适应视图聚合功能，能够实现稠密点云的精确、完整重建。与以往基于深度学习的MVS方法中使用均方方差来生成成本体积不同，我们的\textbf{VA-MVSNet}通过引入两种新的自适应视图聚合：像素视图聚合和体素视图聚合，以较小的额外内存消耗来合并不同视图中的成本方差。为了进一步提高三维点云重建的鲁棒性和完整性，我们扩展了VA-MVSNet与金字塔多尺度图像输入为\textbf{PVA-MVSNet}，其中多度量约束被利用来聚合可靠的深度估计在较粗的尺度，以填充在较细的尺度不匹配的区域。实验结果表明，我们的方法建立了一个新的国家的最先进的\textsl{\textbf{DTU}}数据集的完整性和整体质量的显着改善，并通过实现与国家的最先进的方法在\textsl{\textbf{Tanks and Temples}}基准相当的性能具有很强的泛化能力。我们的代码库位于\hyperlink{https://github.com/yhw-yhw/PVAMVSNet}{https://github.com/yhw-yhw/PVAMVSNet}
SMFormer: Empowering Self-supervised Stereo Matching via Foundation Models and Data Augmentation
Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；Hong Kong Generative AI Research and Development Center, The Hong Kong University of Science and Technology（香港科技大学香港生成式人工智能研发中心）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Electronics and Communication Engineering, the Shenzhen Campus of Sun Yat-sen University, Sun Yat-sen University（中山大学深圳校区电子与通信工程学院）
