While deep learning and deep reinforcement learning (RL) systems have demonstrated impressive results in domains such as image classification, game playing, and robotic control, data efficiency remains a major challenge. Multi-task learning has emerged as a promising approach for sharing structure across multiple tasks to enable more efficient learning. However, the multi-task setting presents a number of optimization challenges, making it difficult to realize large efficiency gains compared to learning tasks independently. The reasons why multi-task learning is so challenging compared to single-task learning are not fully understood. In this work, we identify a set of three conditions of the multi-task optimization landscape that cause detrimental gradient interference, and develop a simple yet general approach for avoiding such interference between task gradients. We propose a form of gradient surgery that projects a task's gradient onto the normal plane of the gradient of any other task that has a conflicting gradient. On a series of challenging multi-task supervised and multi-task RL problems, this approach leads to substantial gains in efficiency and performance. Further, it is model-agnostic and can be combined with previously-proposed multi-task architectures for enhanced performance.
Beyond scalar losses: calibrating segmentation models via gradient vector field surgery
School of Computation, Information and Technology, TUM(慕尼黑工业大学计算、信息与技术学院);Munich Center for Machine Learning(慕尼黑机器学习中心);Department of Radiology, Weill Cornell Medicine(威尔康乃尔医学院放射科);School of Medicine and Health, TUM University Hospital(慕尼黑工业大学医院医学与健康学院);Cornell Tech(康奈尔科技学院);Department of Computing, Imperial College London(伦敦帝国理工学院计算系)
GS-EMA: Integrating Gradient Surgery Exponential Moving Average with Boundary-Aware Contrastive Learning for Enhanced Domain Generalization in Aneurysm Segmentation
University of Leeds(利兹大学);University of Manchester(曼彻斯特大学)
Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery
Institute of Artificial Intelligence, Beihang University(北航人工智能研究院);Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University(北京未来区块链与隐私计算先进创新中心);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Amsterdam(阿姆斯特丹大学);Tsinghua University(清华大学);YanTron Technology Co.Ltd(燕tron技术有限公司)
Spectral Surgery: Training-Free Refinement of LoRA via Gradient-Guided Singular Value Reweighting
School of Computing(计算学院);Information Systems, Singapore Management University(信息系统,新加坡管理大学);School of Computing, National University of Singapore(计算学院,新加坡国立大学);State Key Laboratory for Multimedia Information Processing, Peking University(多媒体信息处理国家重点实验室,北京大学)
SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgery
Northeastern University(东北大学);School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区网络与空间科学学院);University of Maryland(马里兰大学);Nanyang Technological University(南洋理工大学)
针对模型合并多任务学习中的表征偏差问题,本文提出 Surgery 和 SurgeryV2 模块,通过逐层对齐合并模型与专家模型的表征,显著缩小与传统多任务学习的性能差距。
Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery
The Chinese University of Hong Kong(香港中文大学);Johns Hopkins University(约翰斯·霍普金斯大学)
自动手术手势识别对于机器人手术中的智能认知辅助至关重要。随着近年来机器人辅助微创手术的发展,可以记录包括手术视频和机器人运动学在内的丰富信息,这些信息为理解手术手势提供了补充知识。然而,现有的方法要么单独采用单模态数据或直接连接多模态表示,这不能充分利用视觉和运动学数据中固有的信息相关性来提高手势识别的准确性。在这方面,我们提出了一种新的多模态关系图网络的在线方法(即,MRG-Net)通过在潜在特征空间中的交互式消息传播来动态地整合视觉和运动学信息。具体来说,我们首先使用时间卷积网络和LSTM单元从视频和运动序列中提取嵌入。接下来,我们识别这些多模态嵌入中的多关系,并通过层次关系图学习模块来利用它们。我们的方法的有效性证明了国家的最先进的结果在公共JIGSAWS数据集,优于目前的单模态和多模态的方法在两个节点和结类型的任务。此外,我们在两个中心的da Vinci Research Kit(dVRK)平台收集的内部视觉运动学数据集上验证了我们的方法,并实现了一致的有希望的性能。
ReCal-Net: Joint Region-Channel-Wise Calibrated Network for Semantic Segmentation in Cataract Surgery Videos
Department of Information Technology, Alpen-Adria-Universität Klagenfurt(克恩滕州阿尔卑斯-亚得里亚大学信息技术系);Department of Ophthalmology, Klinikum Klagenfurt(克拉根福医院眼科)