Tremendous progress has been made in deep stereo matching to excel on benchmark datasets through per-domain fine-tuning. However, achieving strong zero-shot generalization - a hallmark of foundation models in other computer vision tasks - remains challenging for stereo matching. We introduce FoundationStereo, a foundation model for stereo depth estimation designed to achieve strong zero-shot generalization. To this end, we first construct a large-scale (1M stereo pairs) synthetic training dataset featuring large diversity and high photorealism, followed by an automatic self-curation pipeline to remove ambiguous samples. We then design a number of network architecture components to enhance scalability, including a side-tuning feature backbone that adapts rich monocular priors from vision foundation models to mitigate the sim-to-real gap, and long-range context reasoning for effective cost volume filtering. Together, these components lead to strong robustness and accuracy across domains, establishing a new standard in zero-shot stereo depth estimation. Project page: https://nvlabs.github.io/FoundationStereo/
We propose a fast and accurate method of 6D object pose estimation for bin-picking of mechanical parts by a robot manipulator. We extend the single-shot approach to stereo vision by application of attention architecture. Our convolutional neural network model regresses to object locations and rotations from either a left image or a right image without depth information. Then, a stereo feature matching module, designated as Stereo Grid Attention, generates stereo grid matching maps. The important point of our method is only to calculate disparity of the objects found by the attention from stere…
Stereo Anywhere: Robust Zero-Shot Deep Stereo Matching Even Where Either Stereo or Mono Fail
Advanced Research Center on Electronic System (ARCES)（电子系统高级研究中心）；Department of Computer Science and Engineering (DISI)（计算机科学与工程系）；University of Bologna（博洛尼亚大学）
Stereo-Planner: Learning Stereo-to-Trajectory Planning for Zero-Shot Sim-to-Real Quadrotor Navigation
School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）；School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）
ShotcreteDepth: A Bi-modal Dataset for Robust Robotic Depth Perception in Shotcrete Construction Environments
Technical University of Denmark, Department of Electrical and Photonics Engineering（丹麦技术大学电气与光子工程系）；Pioneer Centre for AI（人工智能先锋中心）；Christiansen & Essenbæk A/S（Christiansen & Essenbæk 公司）
University of Innsbruck（因斯布鲁克大学）；LEC GmbH（LEC有限公司）；INNIO Jenbacher GmbH & Co OG（INNIO颜巴赫两合公司）
Real-time estimation of actual object depth is an essential module for various autonomous system tasks such as 3D reconstruction, scene understanding and condition assessment. During the last decade of machine learning, extensive deployment of deep learning methods to computer vision tasks has yielded approaches that succeed in achieving realistic depth synthesis out of a simple RGB modality. Most of these models are based on paired RGB-depth data and/or the availability of video sequences and stereo images. The lack of sequences, stereo data and RGB-depth pairs makes depth estimation a fully …
Generalization Evaluation of Deep Stereo Matching Methods for UAV-Based Forestry Applications
Centre for Data Science and Artificial Intelligence（数据科学与人工智能中心）；Victoria University of Wellington（惠灵顿维多利亚大学）；Department of Computer Science and Software Engineering（计算机科学与软件工程系）；University of Canterbury（坎特伯雷大学）
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Stereo depth estimation is of great interest for computer vision research. However, existing methods struggles to generalize and predict reliably in hazardous regions, such as large uniform regions. To overcome these limitations, we propose Context Enhanced Path (CEP). CEP improves the generalization and robustness against common failure cases in existing solutions by capturing the long-range global information. We construct our stereo depth estimation model, Context Enhanced Stereo Transformer (CSTR), by plugging CEP into the state-of-the-art stereo depth estimation method Stereo Transformer.…
Single-Shot Metric Depth from Focused Plenoptic Cameras
Institute of Robotics and Mechatronics, German Aerospace Center (DLR)（德国航空航天中心机器人与机电一体化研究所）；I3A, Universidad de Zaragoza（萨拉戈萨大学I3A研究所）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
Diachronic Stereo Matching for Multi-Date Satellite Imagery
IIE, Facultad de Ingeniería, Universidad de la República（乌拉圭国家工程院，工程学院，乌拉圭共和国大学）；Politecnico di Torino（托斯卡纳理工学院）；Eurecat, Centre Tecnologic de Catalunya（欧洲催化中心，加泰罗尼亚技术中心）；AMIAD, Pôle Recherche（AMIAD研究极）；Universite Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli（巴黎萨克雷大学，国家科学研究中心，巴黎萨克雷高等师范学校，博雷利中心）；Institut Universitaire de France（法国国家科学研究中心）
SeasonStereo: Robust Dense Stereo Matching for Multi-Date Satellite Imagery via Generative AI
Eurecat（欧罗卡特技术中心）；IIE, Facultad de Ingeniería, Universidad de la República（乌拉圭共和国大学工程学院IIE）；Universitat Pompeu Fabra（庞培法布拉大学）；Université Paris-Saclay（巴黎萨克雷大学）；CNRS（法国国家科学研究中心）；ENS Paris-Saclay（巴黎萨克雷高等师范学校）；Institut Universitaire de France（法国大学研究院）
Institute for Infocomm Research (IR), A*STAR（A*STAR资讯通信研究院（IR））；Institute for High Performance Computing (IHPC), A*STAR（A*STAR高性能计算研究院（IHPC））
We present KOVIS, a novel learning-based, calibration-free visual servoing method for fine robotic manipulation tasks with eye-in-hand stereo camera system. We train the deep neural network only in the simulated environment; and the trained model could be directly used for real-world visual servoing tasks. KOVIS consists of two networks. The first keypoint network learns the keypoint representation from the image using with an autoencoder. Then the visual servoing network learns the motion based on keypoints extracted from the camera image. The two networks are trained end-to-end in the simula…
MonSter++: Unified Stereo Matching, Multi-view Stereo, and Real-time Stereo with Monodepth Priors
School of Electronic Information and Communications, Huazhong University of Science and Technology（电子信息与通讯学院，华中科技大学）；Meta；AI Chip Center, Hong Kong University of Science and Technology（香港科技大学人工智能芯片中心）；Nanjing Forestry University（南京林业大学）；Autel Robotics
CogStereo: Neural Stereo Matching with Implicit Spatial Cognition Embedding
Robotics and Computer Vision (RCV) Laboratory, Southern University of Science and Technology (SUSTech)（机器人与计算机视觉实验室，南方科技大学）；International Digital Economy Academy (IDEA)（国际数字经济学院）；School of Automation Science and Engineering, Xi’an Jiaotong University (XJTU)（自动化科学与工程学院，西安交通大学）
MV-dVRK: A Multi-Viewpoint Benchmark for Spatial Surgical Perception
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zürich（苏黎世联邦理工学院）；Erbe Group（爱尔博集团）；Tübingen University Hospital（蒂宾根大学医院）；Johns Hopkins University（约翰霍普金斯大学）
Existing approaches for multi-view multi-person 3D pose estimation explicitly establish cross-view correspondences to group 2D pose detections from multiple camera views and solve for the 3D pose estimation for each person. Establishing cross-view correspondences is challenging in multi-person scenes, and incorrect correspondences will lead to sub-optimal performance for the multi-stage pipeline. In this work, we present our multi-view 3D pose estimation approach based on plane sweep stereo to jointly address the cross-view fusion and 3D pose reconstruction in a single shot. Specifically, we p…
Towards Gold-Standard Depth Estimation for Tree Branches in UAV Forestry: Benchmarking Deep Stereo Matching Methods
Centre for Data Science and Artificial Intelligence（数据科学与人工智能中心）；Victoria University of Wellington（惠灵顿维多利亚大学）；Department of Computer Science and Software Engineering（计算机科学与软件工程系）；University of Canterbury（坎特伯雷大学）
German Aerospace Center (DLR), Institute of Space Research（德国航空航天中心（DLR）空间研究所）；Faculty of Geo-Information Science and Earth Observation (ITC), University of Twente（代尔夫特理工大学地理信息科学与地球观测学院）
This paper presents DeepShadow, a one-shot method for recovering the depth map and surface normals from photometric stereo shadow maps. Previous works that try to recover the surface normals from photometric stereo images treat cast shadows as a disturbance. We show that the self and cast shadows not only do not disturb 3D reconstruction, but can be used alone, as a strong learning signal, to recover the depth map and surface normals. We demonstrate that 3D reconstruction from shadows can even outperform shape-from-shading in certain cases. To the best of our knowledge, our method is the first…
3D photography is a new medium that allows viewers to more fully experience a captured moment. In this work, we refer to a 3D photo as one that displays parallax induced by moving the viewpoint (as opposed to a stereo pair with a fixed viewpoint). 3D photos are static in time, like traditional photos, but are displayed with interactive parallax on mobile or desktop screens, as well as on Virtual Reality devices, where viewing it also includes stereo. We present an end-to-end system for creating and viewing 3D photos, and the algorithmic and design choices therein. Our 3D photos are captured in…
NeVStereo: A NeRF-Driven NVS-Stereo Architecture for High-Fidelity 3D Tasks
University of Washington（华盛顿大学）；University of Southern California（南加州大学）；CUHK-Shenzhen（香港中文大学（深圳））；USC Institute for Creative Technologies（南加州大学创意技术研究所）
More than Segmentation: Benchmarking SAM 3 for Segmentation, 3D Perception, and Reconstruction in Robotic Surgery
The Chinese University of Hong Kong（香港中文大学）；The Hong Kong University of Science and Technology（香港科学与技术大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Technical University of Munich（慕尼黑技术大学）
PLANA3R: Zero-shot Metric Planar 3D Reconstruction via Feed-Forward Planar Splatting
The Hong Kong University of Science and Technology（香港科技大学）；Ant Group（蚂蚁集团）；Wuhan University（武汉大学）；Zhejiang University（浙江大学）；The Pennsylvania State University（宾夕法尼亚州立大学）
StereoFactory: A Unified Merging Framework for Robust Stereo Matching
School of Computer Science, Wuhan University（武汉大学计算机学院）；D-Star Robotics；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
Robust Single-shot Structured Light 3D Imaging via Neural Feature Decoding
Wangxuan Institute of Computer Technology, Peking University（北京大学王轩计算机技术研究院）；School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；Yuanpei College, Peking University（北京大学元培学院）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；College of Future Technology, Peking University（北京大学未来技术学院）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）
Over the last decade, one of the most relevant public datasets for evaluating odometry accuracy is the KITTI dataset. Beside the quality and rich sensor setup, its success is also due to the online evaluation tool, which enables researchers to benchmark and compare algorithms. The results are evaluated on the test subset solely, without any knowledge about the ground truth, yielding unbiased, overfit free and therefore relevant validation for robot localization based on cameras, 3D laser or combination of both. However, as any sensor setup, it requires prior calibration and rectified stereo im…
SurgPose: Generalisable Surgical Instrument Pose Estimation using Zero-Shot Learning and Stereo Vision
Department of Computing, Imperial College London（帝国理工学院计算系）；Hamlyn Centre for Robotic Surgery, Department of Surgery and Cancer, Imperial College London（帝国理工学院外科与癌症部门机器人手术中心）
Stereo Anything: Unifying Zero-shot Stereo Matching with Large-Scale Mixed Data
School of Computer Science, Wuhan University（武汉大学计算机科学学院）；Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Bologna（博洛尼亚大学）；Tsinghua University（清华大学）；Waytous Inc.（waytous公司）
Adapting a Foundation Model for Lunar Surface Height Estimation
University of Technology of Troyes（特鲁瓦技术大学）；Hochschule Darmstadt（达姆施塔特应用科学大学）；European Space Agency（欧洲空间局）；Technische Hochschule Würzburg-Schweinfurt（维尔茨堡-施韦因富特应用科学大学）
Geometry Reinforced Efficient Attention Tuning Equipped with Normals for Robust Stereo Matching
Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；Department of Computer Science, Southern Methodist University（南卫理公会大学计算机科学系）；Hon Hai Research Institute（鸿海研究院）
