In real-life applications, machine learning models often face scenarios where there is a change in data distribution between training and test domains. When the aim is to make predictions on distributions different from those seen at training, we incur in a domain generalization problem. Methods to address this issue learn a model using data from multiple source domains, and then apply this model to the unseen target domain. Our hypothesis is that when training with multiple domains, conflicting gradients within each mini-batch contain information specific to the individual domains which is irrelevant to the others, including the test domain. If left untouched, such disagreement may degrade generalization performance. In this work, we characterize the conflicting gradients emerging in domain shift scenarios and devise novel gradient agreement strategies based on gradient surgery to alleviate their effect. We validate our approach in image classification tasks with three multi-domain datasets, showing the value of the proposed agreement strategy in enhancing the generalization capability of deep learning models in domain shift scenarios.
Spectral Surgery: Training-Free Refinement of LoRA via Gradient-Guided Singular Value Reweighting
School of Computing(计算学院);Information Systems, Singapore Management University(信息系统,新加坡管理大学);School of Computing, National University of Singapore(计算学院,新加坡国立大学);State Key Laboratory for Multimedia Information Processing, Peking University(多媒体信息处理国家重点实验室,北京大学)
Challenges in Multi-centric Generalization: Phase and Step Recognition in Roux-en-Y Gastric Bypass Surgery
University of Basel(巴塞尔大学);University of Strasbourg(斯特拉斯堡大学);University of Verona(维罗纳大学);University Hospital of Strasbourg(斯特拉斯堡大学医院);Inselspital Bern University Hospital(伯尔尼大学医院);IRCAD France(法国消化外科和腹腔镜影像研究所)
GS-EMA: Integrating Gradient Surgery Exponential Moving Average with Boundary-Aware Contrastive Learning for Enhanced Domain Generalization in Aneurysm Segmentation
University of Leeds(利兹大学);University of Manchester(曼彻斯特大学)
Beyond scalar losses: calibrating segmentation models via gradient vector field surgery
School of Computation, Information and Technology, TUM(慕尼黑工业大学计算、信息与技术学院);Munich Center for Machine Learning(慕尼黑机器学习中心);Department of Radiology, Weill Cornell Medicine(威尔康乃尔医学院放射科);School of Medicine and Health, TUM University Hospital(慕尼黑工业大学医院医学与健康学院);Cornell Tech(康奈尔科技学院);Department of Computing, Imperial College London(伦敦帝国理工学院计算系)
Domain Generalization via Pareto Optimal Gradient Matching
School of Computer Science;Statistics, Trinity College Dublin, Dublin, Ireland;Pusan National University, Busan, South Korea;SoICT, Hanoi University of Science;College of Engineering \& Computer Science, Vin University, Hanoi, Vietnam
OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding
Monash University(莫纳什大学);Airdoc-Monash Research(Airdoc-莫纳什研究所);Airdoc;Bosch Corporate Research(博世企业研究院);Cornell University(康奈尔大学);King’s College London(伦敦国王学院);Institute for Infocomm Research, A*STAR(新加坡资讯通信研究院(新加坡科技研究局下属));City University of Macau(澳门城市大学);Eye Hospital, Wenzhou Medical University(温州医科大学附属眼视光医院)
University of Oxford(牛津大学);Facebook Reality Labs(Facebook现实实验室);The University of Edinburgh(爱丁堡大学);The Alan Turing Institute(艾伦·图灵研究所);Facebook AI Research(Facebook人工智能研究院)
Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery
Institute of Artificial Intelligence, Beihang University(北航人工智能研究院);Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University(北京未来区块链与隐私计算先进创新中心);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Amsterdam(阿姆斯特丹大学);Tsinghua University(清华大学);YanTron Technology Co.Ltd(燕tron技术有限公司)
CataractSAM-2: A Domain-Adapted Model for Anterior Segment Surgery Segmentation and Scalable Ground-Truth Annotation
Thomas Jefferson High School for Science and Technology(托马斯·杰弗逊高中科学与技术学校);Mass Eye and Ear, Harvard Medical School(麻省眼耳研究所,哈佛医学院);Harvard Ophthalmology AI Lab, Schepens Eye Research Institute of Mass Eye and Ear, Harvard Medical School(哈佛眼科学人工智能实验室,麻省眼耳研究所的谢普恩斯眼研究学院,哈佛医学院)
Loss Gradient Gaussian Width based Generalization and Optimization Guarantees
Siebel School of Computing and Data Science(西贝尔计算与数据科学学院);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Minnesota, Twin Cities(明尼苏达大学双城分校)
Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learning
Tsinghua University(清华大学)
如何训练深度神经网络(DNN)以良好地泛化是深度学习的核心问题,特别是对于当今严重过参数化的网络。在本文中,我们提出了一种有效的方法来提高模型的推广,额外的惩罚梯度范数的损失函数在优化过程中。我们证明了限制损失函数的梯度范数可以帮助引导优化器找到平坦的最小值。我们利用一阶近似来有效地实现相应的梯度,以很好地适应梯度下降框架。在我们的实验中,我们证实,当使用我们的方法时,各种模型的泛化性能可以在不同的数据集上得到提高。此外,我们表明,最近的锐度感知最小化方法(Foret et al.,2021)是我们方法的一个特殊但不是最好的情况,我们方法的最好情况可以在这些任务上提供新的最先进的性能。代码可以在{https://github.com/zhaoyang-0204/gnp}上找到。
Stability and Generalization of the Decentralized Stochastic Gradient Descent
College of Computer, National University of Defense Technology(国防科技大学计算机学院);Scientific Computing & Imaging Institute, University of Utah(犹他大学科学计算与成像研究所)
Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernel
University of California San Diego(加州大学圣地亚哥分校);University of California Berkeley(加州大学伯克利分校);ICSI(国际计算机科学研究所);RIKEN AIP(理化学研究所Advanced Institute for Scientific Research);University of Sydney(悉尼大学);University of Tokyo(东京大学)