Vision-based segmentation of the robotic tool during robot-assisted surgery enables downstream applications, such as augmented reality feedback, while allowing for inaccuracies in robot kinematics. With the introduction of deep learning, many methods were presented to solve instrument segmentation directly and solely from images. While these approaches made remarkable progress on benchmark datasets, fundamental challenges pertaining to their robustness remain. We present CaRTS, a causality-driven robot tool segmentation algorithm, that is designed based on a complementary causal model of the robot tool segmentation task. Rather than directly inferring segmentation masks from observed images, CaRTS iteratively aligns tool models with image observations by updating the initially incorrect robot kinematic parameters through forward kinematics and differentiable rendering to optimize image feature similarity end-to-end. We benchmark CaRTS with competing techniques on both synthetic as well as real data from the dVRK, generated in precisely controlled scenarios to allow for counterfactual synthesis. On training-domain test data, CaRTS achieves a Dice score of 93.4 that is preserved well (Dice score of 91.8) when tested on counterfactually altered test data, exhibiting low brightness, smoke, blood, and altered background patterns. This compares favorably to Dice scores of 95.0 and 86.7, respectively, of the SOTA image-based method. Future work will involve accelerating CaRTS to achieve video framerate and estimating the impact occlusion has in practice. Despite these limitations, our results are promising: In addition to achieving high segmentation accuracy, CaRTS provides estimates of the true robot kinematics, which may benefit applications such as force estimation. Code is available at: https://github.com/hding2455/CaRTS
TrueRMA: Learning Fast and Smooth Robot Trajectories with Recursive Midpoint Adaptations in Cartesian Space
Institute for Anthropomatics and Robotics – Intelligent Process Automation and Robotics (IAR-IPR)（拟人技术与机器人研究所——智能过程自动化与机器人（IAR-IPR））；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院（KIT））
Inverse Kinematics with Forward Dynamics Solvers for Sampled Motion Tracking
FZI Research Center for Information Technology（FZI信息技术研究中心）；IAR Institute for Anthropomatics and Robotics（IAR拟人系统与机器人研究所）；KIT Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
KiGRAS: Kinematic-Driven Generative Model for Realistic Agent Simulation
University of Science and Technology of China（中国科学技术大学）；Mach Drive（马赫动力）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
