Conventional fine-tuning on domain-specific datasets can inadvertently alter a model's pretrained multimodal priors, leading to reduced generalization. To address this, we propose Chain-of-Adaptation (CoA), an adaptation framework designed to integrate domain knowledge while maintaining the model's inherent reasoning and perceptual capabilities. CoA introduces a structured reasoning format that enhances domain alignment without sacrificing general multimodal competence by reinforcement learning. Experiments on standard surgical benchmarks, under both in-distribution and out-of-distribution settings, demonstrate that CoA achieves higher accuracy, stronger generalization, and more stable behavior than supervised fine-tuning. Furthermore, ablation studies confirm that CoA effectively preserves the model's core visual-language abilities, providing a reliable pathway for domain specialization in VLMs.
University College London（伦敦大学学院）；University Hospital Leuven（鲁汶大学医院）
In minimally invasive surgery, surgical workflow segmentation from video analysis is a well studied topic. The conventional approach defines it as a multi-class classification problem, where individual video frames are attributed a surgical phase label. We introduce a novel reinforcement learning formulation for offline phase transition retrieval. Instead of attempting to classify every video frame, we identify the timestamp of each phase transition. By construction, our model does not produce spurious and noisy phase transitions, but contiguous phase blocks. We investigate two different confi…
Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learning
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；Department of Thoracic Surgery, the Seventh Affiliated Hospital, Sun Yat-sen University（中山大学第七附属医院胸外科部门）；Bioengineering/Imperial-X, Imperial College London（生物工程/Imperial-X，帝国理工学院伦敦分校）；ROAS Thrust, Hong Kong University of Science and Technology (Guangzhou)（ROAS项目，香港科技大学（广州））；Department of Electronic and Computer Engineering, Hong Kong SAR（香港特别行政区电子与计算机工程系）
DARIL: When Imitation Learning outperforms Reinforcement Learning in Surgical Action Planning
Surgical and Interventional Engineering, King's College London, London, UK（外科与介入工程，伦敦国王学院，伦敦，英国）；Interventional Engineering, King's College London, London, UK（介入工程，伦敦国王学院，伦敦，英国）
该研究在 CholecT50 上比较手术动作规划中的 IL 与 RL，提出 DARIL 基线，发现其显著优于三种 RL 方法，并揭示专家测试分布对 IL 的系统性偏好。
Task automation of surgical robot has the potentials to improve surgical efficiency. Recent reinforcement learning (RL) based approaches provide scalable solutions to surgical automation, but typically require extensive data collection to solve a task if no prior knowledge is given. This issue is known as the exploration challenge, which can be alleviated by providing expert demonstrations to an RL agent. Yet, how to make effective use of demonstration data to improve exploration efficiency still remains an open challenge. In this work, we introduce Demonstration-guided EXploration (DEX), an e…
Institute for Intelligent Systems Research and Innovation, Deakin University（迪肯大学智能系统研究与创新研究所）；Imperial College London（伦敦帝国理工学院）
Surgeons normally need surgical scissors and tissue grippers to cut through a deformable surgical tissue. The cutting accuracy depends on the skills to manipulate these two tools. Such skills are part of basic surgical skills training as in the Fundamentals of Laparoscopic Surgery. The gripper is used to pinch a point on the surgical sheet and pull the tissue to a certain direction to maintain the tension while the scissors cut through a trajectory. As the surgical materials are deformable, it requires a comprehensive tensioning policy to yield appropriate tensioning direction at each step of …
Imperial College London（伦敦帝国理工学院）；University of Bristol（布里斯托尔大学）
In recent decades, the tremendous benefits surgical robots have brought to surgeons and patients have been witnessed. With the dexterous operation and the great precision, surgical robots can offer patients less recovery time and less hospital stay. However, the controls for current surgical robots in practical usage are fully carried out by surgeons via teleoperation. During the surgery process, there exists a lot of repetitive but simple manipulation, which can cause unnecessary fatigue to the surgeons. In this paper, we proposed a deep reinforcement learning-based semi-autonomous control fr…
The Chinese University of Hong Kong（香港中文大学）；CUHK T Stone Robotics Institute（香港中文大学天石机器人研究所）
Automatic surgical gesture recognition is fundamental for improving intelligence in robot-assisted surgery, such as conducting complicated tasks of surgery surveillance and skill evaluation. However, current methods treat each frame individually and produce the outcomes without effective consideration on future information. In this paper, we propose a framework based on reinforcement learning and tree search for joint surgical gesture segmentation and classification. An agent is trained to segment and classify the surgical video in a human-like manner whose direct decisions are re-considered b…
Over its lifetime, a reinforcement learning agent is often tasked with different tasks. How to efficiently adapt a previously learned control policy from one task to another, remains an open research question. In this paper, we investigate how instructions formulated in natural language can enable faster and more effective task adaptation. This can serve as the basis for developing language instructed skills, which can be used in a lifelong learning setting. Our method is capable of assessing, given a set of developed base control policies, which policy will adapt best to a new unseen task.
fine-CLIP: Enhancing Zero-Shot Fine-Grained Surgical Action Recognition with Vision-Language Models
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；ICube（ICube实验室）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）
Deep Reinforcement Learning (DRL) aims to create intelligent agents that can learn to solve complex problems efficiently in a real-world environment. Typically, two learning goals: adaptation and generalization are used for baselining DRL algorithm's performance on different tasks and domains. This paper presents a survey on the recent developments in DRL-based approaches for adaptation and generalization. We begin by formulating these goals in the context of task and domain. Then we review the recent works under those approaches and discuss future research directions through which DRL algorit…
Reinforcement learning has shown a wide usage in robotics tasks, such as insertion and grasping. However, without a practical sim2real strategy, the policy trained in simulation could fail on the real task. There are also wide researches in the sim2real strategies, but most of those methods rely on heavy image rendering, domain randomization training, or tuning. In this work, we solve the insertion task using a pure visual reinforcement learning solution with minimum infrastructure requirement. We also propose a novel sim2real strategy, Real2Sim, which provides a novel and easier solution in p…
Artificial Intelligence for Climate Adaptation: Reinforcement Learning for Climate Change-Resilient Transport
Department of Technology, Management and Economics, Technical University of Denmark（技术、管理与经济学系，丹麦技术大学）；J.E. Cairnes School of Business and Economics, University of Galway（J.E. Cairnes 商业与经济学学院，Galway大学）
Developments in reinforcement learning (RL) have allowed algorithms to achieve impressive performance in highly complex, but largely static problems. In contrast, biological learning seems to value efficiency of adaptation to a constantly-changing world. Here we build on a recently-proposed neuronal learning rule that assumes each neuron can optimize its energy balance by predicting its own future activity. That assumption leads to a neuronal learning rule that uses presynaptic input to modulate prediction error. We argue that an analogous RL rule would use action probability to modulate rewar…
University of the Basque Country UPV/EHU（巴斯克大学（UPV/EHU））；IKERBASQUE, Basque Foundation for Science（巴斯克科学基金会（IKERBASQUE））；Shanghai University（上海大学）；International Center of Quantum Artificial Intelligence for Science and Technology (QuArtist)（国际科学技术量子人工智能中心（QuArtist））；IQM（IQM公司）；Universidad de Sevilla（塞维利亚大学）
We present an experimental realization of a measurement-based adaptation protocol with quantum reinforcement learning in a Rigetti cloud quantum computer. The experiment in this few-qubit superconducting chip faithfully reproduces the theoretical proposal, setting the first steps towards a semiautonomous quantum agent. This experiment paves the way towards quantum reinforcement learning with superconducting circuits.
Automatic surgical activity recognition enables more intelligent surgical devices and a more efficient workflow. Integration of such technology in new operating rooms has the potential to improve care delivery to patients and decrease costs. Recent works have achieved a promising performance on surgical activity recognition; however, the lack of generalizability of these models is one of the critical barriers to the wide-scale adoption of this technology. In this work, we study the generalizability of surgical activity recognition models across operating rooms. We propose a new domain adaptati…
Deep reinforcement learning agents have recently been successful across a variety of discrete and continuous control tasks; however, they can be slow to train and require a large number of interactions with the environment to learn a suitable policy. This is borne out by the fact that a reinforcement learning agent has no prior knowledge of the world, no pre-existing data to depend on and so must devote considerable time to exploration. Transfer learning can alleviate some of the problems by leveraging learning done on some source task to help learning on some target task. Our work presents an…
On the Robustness of Temporal Vision-Language Models for Surgical Endoscopy Videos
Indian Institute of Technology Delhi（印度理工学院德里分校）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）；Birmingham City University（伯明翰城市大学）；University Hospitals Birmingham（伯明翰大学医院）；Khalifa University（哈利法大学）；German Cancer Research Center (DKFZ)（德国癌症研究中心）
Continual Knowledge Adaptation for Reinforcement Learning
South China University of Technology（南方科技大学）；Pazhou Laboratory（Pazhou实验室）；Chongqing University of Posts and Telecommunications（重庆邮电大学）；Key Laboratory of Big Data and Intelligent Robot, Ministry of Education（大数据与智能机器人重点实验室，教育部）
DenseTRF: Texture-Aware Unsupervised Representation Adaptation for Surgical Scene Dense Prediction
GRASP Laboratory, University of Pennsylvania（宾夕法尼亚大学GRASP实验室）；PCASO Laboratory, Department of Surgery, University of Pennsylvania（宾夕法尼亚大学外科PCASO实验室）；Department of Computer and Information Science, University of Pennsylvania（宾夕法尼亚大学计算机与信息科学系）
We introduce a new algorithm for multi-objective reinforcement learning (MORL) with linear preferences, with the goal of enabling few-shot adaptation to new tasks. In MORL, the aim is to learn policies over multiple competing objectives whose relative importance (preferences) is unknown to the agent. While this alleviates dependence on scalar reward design, the expected return of a policy can change significantly with varying preferences, making it challenging to learn a single model to produce optimal policies under different preference conditions. We propose a generalized version of the Bell…
University of California, Los Angeles（加州大学洛杉矶分校）；Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Helixon Ltd.（Helixon有限公司）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）
Learning new task-specific skills from a few trials is a fundamental challenge for artificial intelligence. Meta reinforcement learning (meta-RL) tackles this problem by learning transferable policies that support few-shot adaptation to unseen tasks. Despite recent advances in meta-RL, most existing methods require the access to the environmental reward function of new tasks to infer the task objective, which is not realistic in many practical applications. To bridge this gap, we study the problem of few-shot adaptation in the context of human-in-the-loop reinforcement learning. We develop a m…
Surgical-DINO: Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgery
The Chinese University of Hong Kong（香港中文大学）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London（伦敦大学学院惠康/工程与物理科学研究委员会介入与外科科学中心）；National University of Singapore（新加坡国立大学）
This paper covers n-tuple-based reinforcement learning (RL) algorithms for games. We present new algorithms for TD-, SARSA- and Q-learning which work seamlessly on various games with arbitrary number of players. This is achieved by taking a player-centered view where each player propagates his/her rewards back to previous rounds. We add a new element called Final Adaptation RL (FARL) to all these algorithms. Our main contribution is that FARL is a vitally important ingredient to achieve success with the player-centered view in various games. We report results on seven board games with 1, 2 and…
Despite the recent success of deep reinforcement learning (RL), domain adaptation remains an open problem. Although the generalization ability of RL agents is critical for the real-world applicability of Deep RL, zero-shot policy transfer is still a challenging problem since even minor visual changes could make the trained agent completely fail in the new task. To address this issue, we propose a two-stage RL agent that first learns a latent unified state representation (LUSR) which is consistent across multiple domains in the first stage, and then do RL training in one source domain based on …
This paper introduces Fast Linearized Adaptive Policy (FLAP), a new meta-reinforcement learning (meta-RL) method that is able to extrapolate well to out-of-distribution tasks without the need to reuse data from training, and adapt almost instantaneously with the need of only a few samples during testing. FLAP builds upon the idea of learning a shared linear representation of the policy so that when adapting to a new task, it suffices to predict a set of linear weights. A separate adapter network is trained simultaneously with the policy such that during adaptation, we can directly use the adap…
Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deployment
Lyles School of Civil and Construction Engineering, Purdue University, West Lafayette, USA（普渡大学土木与建设工程学院）；Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, USA（普渡大学埃尔莫尔家庭电气与计算机工程学院）
Joint replacement is the most common inpatient surgical treatment in the US. We investigate the clinical pathway optimization for knee replacement, which is a sequential decision process from onset to recovery. Based on episodic claims from previous cases, we view the pathway optimization as an intelligence crowdsourcing problem and learn the optimal decision policy from data by imitating the best expert at every intermediate state. We develop a reinforcement learning-based pipeline that uses value iteration, state compression and aggregation learning, kernel representation and cross validatio…
Intuitive, Inc.（直观医疗公司）；Konica Minolta, Inc.（柯尼卡美能达公司）；University College London（伦敦大学学院）；University of Genoa（热那亚大学）；University of Burgundy（勃艮第大学）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）
Surgical data science is revolutionizing minimally invasive surgery by enabling context-aware applications. However, many challenges exist around surgical data (and health data, more generally) needed to develop context-aware models. This work - presented as part of the Endoscopic Vision (EndoVis) challenge at the Medical Image Computing and Computer Assisted Intervention (MICCAI) 2020 conference - seeks to explore the potential for visual domain adaptation in surgery to overcome data privacy concerns. In particular, we propose to use video from virtual reality (VR) simulations of surgical exe…
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；ICube（ICube研究中心）；UMR7357（法国大学-研究中心7357）；IHU Strasbourg（斯特拉斯堡IHU医院）；University Digestive Health Care Center – Clarunis（大学消化健康中心 – Clarunis）；Ludwig Maximilian University of Munich（慕尼黑路易斯·马克西米利安大学）；Chinese University of Hong Kong（香港中文大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）
Hierarchical reinforcement learning is a promising approach to tackle long-horizon decision-making problems with sparse rewards. Unfortunately, most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in a new task. Leaving the skills fixed can lead to significant sub-optimality in the transfer setting. In this work, we propose a novel algorithm to discover a set of skills, and continuously adapt them along with the higher level even when training on a new task. Our main contributions are two-fold. First, we derive a new …
Arizona State University（亚利桑那州立大学）；Facebook AI Research（脸书人工智能研究院）
Humans and animals are capable of quickly learning new behaviours to solve new tasks. Yet, we often forget that they also rely on a highly specialized morphology that co-adapted with motor control throughout thousands of years. Although compelling, the idea of co-adapting morphology and behaviours in robots is often unfeasible because of the long manufacturing times, and the need to re-design an appropriate controller for each morphology. In this paper, we propose a novel approach to automatically and efficiently co-adapt a robot morphology and its controller. Our approach is based on recent a…
Recognition of surgical gesture is crucial for surgical skill assessment and efficient surgery training. Prior works on this task are based on either variant graphical models such as HMMs and CRFs, or deep learning models such as Recurrent Neural Networks and Temporal Convolutional Networks. Most of the current approaches usually suffer from over-segmentation and therefore low segment-level edit scores. In contrast, we present an essentially different methodology by modeling the task as a sequential decision-making process. An intelligent agent is trained using reinforcement learning with hier…
