Operating effectively in novel real-world environments requires robotic systems to estimate and interact with previously unseen objects. Current state-of-the-art models address this challenge by using large amounts of training data and test-time samples to build black-box scene representations. In this work, we introduce a differentiable neuro-graphics model that combines neural foundation models with physics-based differentiable rendering to perform zero-shot scene reconstruction and robot grasping without relying on any additional 3D data or test-time samples. Our model solves a series of constrained optimization problems to estimate physically consistent scene parameters, such as meshes, lighting conditions, material properties, and 6D poses of previously unseen objects from a single RGBD image and bounding boxes. We evaluated our approach on standard model-free few-shot benchmarks and demonstrated that it outperforms existing algorithms for model-free few-shot pose estimation. Furthermore, we validated the accuracy of our scene reconstructions by applying our algorithm to a zero-shot grasping task. By enabling zero-shot, physically-consistent scene reconstruction and grasping without reliance on extensive datasets or test-time sampling, our approach offers a pathway towards more data efficient, interpretable and generalizable robot autonomy in novel environments.
D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Grasping
Physical Superintelligence (PSI) Lab, University of Southern California（南加州大学物理超智能实验室）；Viterbi School of Engineering, University of Southern California（南加州大学韦伯尔工程学院）；Department of EECS, University of California, Berkeley（加州大学伯克利分校电子工程与计算机科学系）
Simultaneous Semantic and Collision Learning for 6-DoF Grasp Pose Estimation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Bytedance AI Lab（字节跳动人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）
Learning Scene-Level Signed Directional Distance Function with Ellipsoidal Priors and Neural Residuals
Department of Electrical and Computer Engineering, University of California San Diego（加州大学圣地亚哥分校电气与计算机工程系）；Brain Corporation（Brain公司）；Robotics Department, University of Michigan（密歇根大学机器人系）
Image-driven Robot Drawing with Rapid Lognormal Movements
Goldsmiths, University of London（伦敦大学金史密斯学院）；Idiap Research Institute and École Polytechnique Fédérale de Lausanne (EPFL)（日内瓦联邦理工学院（EPFL）与IDIAP研究所）；University of Konstanz（康斯坦茨大学）；Ecole Polytechnique de Montréal（蒙特利尔联邦理工学院）
