Laparoscopic Field of View (FOV) control is one of the most fundamental and important components in Minimally Invasive Surgery (MIS), nevertheless, the traditional manual holding paradigm may easily bring fatigue to surgical assistants, and misunderstanding between surgeons also hinders assistants to provide a high-quality FOV. Targeting this problem, we here present a data-driven framework to realize an automated laparoscopic optimal FOV control. To achieve this goal, we offline learn a motion strategy of laparoscope relative to the surgeon's hand-held surgical tool from our in-house surgical videos, developing our control domain knowledge and an optimal view generator. To adjust the laparoscope online, we first adopt a learning-based method to segment the two-dimensional (2D) position of the surgical tool, and further leverage this outcome to obtain its scale-aware depth from dense depth estimation results calculated by our novel unsupervised RoboDepth model only with the monocular camera feedback, hence in return fusing the above real-time 3D position into our control loop. To eliminate the misorientation of FOV caused by Remote Center of Motion (RCM) constraints when moving the laparoscope, we propose a novel distortion constraint using an affine map to minimize the visual warping problem, and a null-space controller is also embedded into the framework to optimize all types of errors in a unified and decoupled manner. Experiments are conducted using Universal Robot (UR) and Karl Storz Laparoscope/Instruments, which prove the feasibility of our domain knowledge and learning enabled framework for automated camera control.
City University of Hong Kong（香港城市大学）；The Chinese University of Hong Kong（香港中文大学）
Automatic laparoscope motion control is fundamentally important for surgeons to efficiently perform operations. However, its traditional control methods based on tool tracking without considering information hidden in surgical scenes are not intelligent enough, while the latest supervised imitation learning (IL)-based methods require expensive sensor data and suffer from distribution mismatch issues caused by limited demonstrations. In this paper, we propose a novel Imitation Learning framework for Laparoscope Control (ILLC) with reinforcement learning (RL), which can efficiently learn the con…
A holistic perception system of internal and external monitoring for ground autonomous vehicles: AutoTRUST paradigm
Industrial Systems Institute, ATHENA Research Center, Patras Science Park, Greece（工业系统研究所，ATHENA研究中心，帕特拉斯科学公园，希腊）；AviSense.AI, Patras Science Park, Greece（AviSense.AI，帕特拉斯科学公园，希腊）；Information Technologies Institute, CERTH, Greece（信息科技研究所，CERTH，希腊）；KIOS Research and Innovation Center of Excellence, University of Cyprus, Cyprus（KIOS研究与创新中心，塞浦路斯大学，塞浦路斯）
University of Genova（热那亚大学）；Italian Institute of Technology（意大利理工学院）
As robotic technology advances, the barriers to the coexistence of humans and robots are slowly coming down. Application domains like elderly care, collaborative manufacturing, collaborative manipulation, etc., are considered the need of the hour, and progress in robotics holds the potential to address many societal challenges. The future socio-technical systems constitute of blended workforce with a symbiotic relationship between human and robot partners working collaboratively. This thesis attempts to address some of the research challenges in enabling human-robot collaboration. In particula…
EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI
The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；The University of Hong Kong（香港大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
EHC-MM: Embodied Holistic Control for Mobile Manipulation
Beijing Samsung Telecom R&D Center（北京三星通信技术研发中心）；National Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）；School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；PKU-WUHAN Institute for Artificial Intelligence（北京大学-武汉人工智能研究院）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Tsinghua University（清华大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
From Prompt to Physical Actuation: Holistic Threat Modeling of LLM-Enabled Robotic Systems
School of Informatics, Computing, and Cyber Systems, Northern Arizona University（信息学、计算与网络系统学院，北亚利桑那大学）；Department of Software Science, Tallinn University of Technology（软件科学系，塔林技术大学）
This paper presents an automated driving system (ADS) data acquisition and processing platform for vehicle trajectory extraction, reconstruction, and evaluation based on connected automated vehicle (CAV) cooperative perception. This platform presents a holistic pipeline from the raw advanced sensory data collection to data processing, which can process the sensor data from multiple CAVs and extract the objects' Identity (ID) number, position, speed, and orientation information in the map and Frenet coordinates. First, the ADS data acquisition and analytics platform are presented. Specifically,…
Breaking the Latency Barrier: Synergistic Perception and Control for High-Frequency 3D Ultrasound Servoing
Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China（香港中文大学电子工程系）；Department of Biomedical Engineering, Tsinghua University, Beijing, China（清华大学生物医学工程系）；Great Bay University, Dongguan, China（东莞大邦大学）
Humans have an innate ability to sense their surroundings, as they can extract the spatial representation from the egocentric perception and form an allocentric semantic map via spatial transformation and memory updating. However, endowing mobile agents with such a spatial sensing ability is still a challenge, due to two difficulties: (1) the previous convolutional models are limited by the local receptive field, thus, struggling to capture holistic long-range dependencies during observation; (2) the excessive computational budgets required for success, often lead to a separation of the mappin…
Data-driven perception approaches are well-established in automated driving systems. In many fields even super-human performance is reached. Unlike prediction and planning approaches, mainly supervised learning algorithms are used for the perception domain. Therefore, a major remaining challenge is the efficient generation of ground truth data. As perception modules are positioned close to the sensor, they typically run on raw sensor data of high bandwidth. Due to that, the generation of ground truth labels typically causes a significant manual effort, which leads to high costs for the labelli…
