The performance of multi-task learning in Convolutional Neural Networks (CNNs) hinges on the design of feature sharing between tasks within the architecture. The number of possible sharing patterns are combinatorial in the depth of the network and the number of tasks, and thus hand-crafting an architecture, purely based on the human intuitions of task relationships can be time-consuming and suboptimal. In this paper, we present a probabilistic approach to learning task-specific and shared representations in CNNs for multi-task learning. Specifically, we propose "stochastic filter groups'' (SFG), a mechanism to assign convolution kernels in each layer to "specialist'' or "generalist'' groups, which are specific to or shared across different tasks, respectively. The SFG modules determine the connectivity between layers and the structures of task-specific and shared representations in the network. We employ variational inference to learn the posterior distribution over the possible grouping of kernels and network parameters. Experiments demonstrate that the proposed method generalises across multiple tasks and shows improved performance over baseline methods.
Super-Generalist: Towards Comprehensive and Accurate Medical Image Understanding via Generalist-Specialist Synergy
DAMO Academy, Alibaba Group（达摩院，阿里巴巴集团）；Hupan Lab（湖畔实验室）；Department of Radiology, Ningbo No. 2 Hospital（宁波市第二医院放射科）；Department of Radiology, Guangdong Provincial People’s Hospital（广东省人民医院放射科）；Department of Radiology, Shanghai Institution of Pancreatic Disease（上海胰腺疾病研究所放射科）；Department of Radiology, Shengjing Hospital of China Medical University（中国医科大学附属盛京医院放射科）；The First Affiliated Hospital, Zhejiang University School of Medicine（浙江大学医学院附属第一医院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imaging
Technical University of Munich (TUM)（慕尼黑工业大学）；TUM University Hospital（慕尼黑工业大学附属医院）；Hasso Plattner Institute for Digital Engineering（哈索·普拉特纳数字工程研究所）；University of Potsdam（波茨坦大学）；Imperial College London（伦敦帝国理工学院）
GSCo: Towards Generalizable AI in Medicine via Generalist-Specialist Collaboration
The Hong Kong University of Science and Technology（香港科技大学）；Harvard University（哈佛大学）；Sun Yat-sen University Cancer Center（中山大学肿瘤防治中心）；Shenzhen People’s Hospital（深圳市人民医院）；Weill（威尔（未完整显示））
From Specialist to Generalist: Unlocking SAM's Learning Potential on Unlabeled Medical Images
Carnegie Mellon University（卡内基梅隆大学）；Industrial University of Ho Chi Minh City（胡志明市工业大学）；University of North Carolina at Charlotte（北卡罗来纳州夏洛特大学）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）
EoS-FM: Can an Ensemble of Specialist Models act as a Generalist Feature Extractor?
IRISA, Université Bretagne Sud, UMR 6074（IRISA，南布列塔尼大学，UMR 6074）；Centre National d’Études Spatiales (CNES)（法国国家空间研究中心（CNES））；UiT The Arctic University of Norway（挪威北极大学）
BRIGHT: A Collaborative Generalist-Specialist Foundation Model for Breast Pathology
Department of Breast Pathology and Laboratory, Tianjin Medical University Cancer Institute & Hospital, National Clinical Research Center for Cancer, Key Laboratory of Breast Cancer Prevention and Therapy, Tianjin Medical University, Ministry of Education, Tianjin’s Clinical Research Center for Cancer, West Huanhu Road, Tianjin, China（天津医科大学肿瘤医院乳腺病理科及实验室，国家癌症临床研究中心，天津医科大学乳腺癌预防与治疗重点实验室，天津医科大学，教育部，天津癌症临床研究中心，西湖南路，天津，中国）；Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application, Guangdong Provincial People’s Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, China（广东省人工智能在医学影像分析与应用重点实验室，广东省人民医院（广东省医学科学院），南方医科大学，广州，中国）
Med-LEGO: Editing and Adapting toward Generalist Medical Image Diagnosis
School of Biomedical Engineering \& State Key Laboratory of Advanced Medical Materials；Devices, ShanghaiTech University, Shanghai, 201210, China Shanghai Clinical Research；Trial Center, Shanghai, 201210, China
Generalist versus Specialist Vision Foundation Models for Ocular Disease and Oculomics
Institute of Ophthalmology, University College London（伦敦大学学院眼科研究所）；Hawkes Institute, University College London（伦敦大学学院霍克斯研究所）；NIHR Biomedical Research Centre at Moorfields Eye Hospital NHS Foundation Trust（莫菲尔德眼科医院 NHS 基金会信托 NIHR 生物医学研究中心）；Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院）；Centre for Innovation and Precision Eye Health, Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院创新与精准眼健康中心）；Singapore Eye Research Institute, Singapore National Eye Centre（新加坡国家眼科中心新加坡眼科研究所）；Chinese University of Hong Kong（香港中文大学）；Shanghai Jiao Tong University（上海交通大学）；MOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院教育部人工智能重点实验室）；Stadtspital Zürich（苏黎世市立医院）；Spross Research Institute（斯普罗斯研究所）；Tsinghua Medicine, Tsinghua University（清华大学清华医学）；Beijing Visual Science and Translational Eye Research Institute, Beijing Tsinghua Changgung Hospital（北京清华长庚医院北京视觉科学与转化眼科研究所）；University College London（伦敦大学学院）
From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuning
Singapore Eye Research Institute, Singapore National Eye Centre（新加坡眼科研究所、新加坡国家眼科中心）；Duke-National University of Singapore Medical School（杜克-新加坡国立大学医学院）；Byers Eye Institute, Stanford University（斯坦福大学拜尔斯眼科研究所）
From Generalist to Specialist: A Context-Fusion Framework for Endoscopic Polyp Reporting with a Frozen VLM
Zhejiang University（浙江大学）；Shanghai Institute for Advanced Study, Zhejiang University（浙江大学上海高等研究院）；Shanghai Key Laboratory of MICCAI（上海市MICCAI重点实验室）；Digital Medical Research Center, School of Basic Medical Sciences, Fudan University（复旦大学基础医学院数字医学研究中心）；Endoscopy Center and Endoscopy Research Institute, Zhongshan Hospital, Fudan University（复旦大学附属中山医院内镜中心及内镜研究所）；Shanghai Collaborative Innovation Center of Endoscopy（上海市内镜协同创新中心）；Alliance Manchester Business School, The University of Manchester（曼彻斯特大学联盟曼彻斯特商学院）
SemiSAM+: Rethinking Semi-Supervised Medical Image Segmentation in the Era of Foundation Models
Artificial Intelligence Innovation and Incubation Institute, Fudan University（复旦大学人工智能创新与孵化研究院）；Shanghai Academy of Artificial Intelligence for Science（上海人工智能科学研究院）；Huashan Hospital, Fudan University（复旦大学附属华山医院）；Human Phenome Institute, Fudan University（复旦大学人类表型组研究院）；School of Information Science and Engineering, Lanzhou University（兰州大学信息科学与工程学院）
Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Grounding
University of California, San Diego（加州大学圣地亚哥分校）；ByteDance（字节跳动）；University of California, Merced（加州大学默塞德分校）；University of Southern California（南加州大学）；University at Buffalo（布法罗大学）；The University of Queensland（昆士兰大学）
Convolutional neural networks (CNNs) apply well with food image recognition due to the ability to learn discriminative visual features. Nevertheless, recognizing distorted images is challenging for existing CNNs. Hence, the study modelled a generalized specialist approach to train a quality resilient ensemble. The approach aids the models in the ensemble framework retain general skills of recognizing clean images and shallow skills of classifying noisy images with one deep expertise area on a particular distortion. Subsequently, a novel data augmentation random quality mixup (RQMixUp) is combi…
Dexterous manipulation of arbitrary objects, a fundamental daily task for humans, has been a grand challenge for autonomous robotic systems. Although data-driven approaches using reinforcement learning can develop specialist policies that discover behaviors to control a single object, they often exhibit poor generalization to unseen ones. In this work, we show that policies learned by existing reinforcement learning algorithms can in fact be generalist when combined with multi-task learning and a well-chosen object representation. We show that a single generalist policy can perform in-hand man…
OneDiff: A Generalist Model for Image Difference Captioning
Institute of Automation, Chinese Academy of Sciences, China（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences, China（中国科学院大学人工智能学院）
University of Southern California（南加州大学）；Google Research（谷歌研究院）
Fine-grained image labels are desirable for many computer vision applications, such as visual search or mobile AI assistant. These applications rely on image classification models that can produce hundreds of thousands (e.g. 100K) of diversified fine-grained image labels on input images. However, training a network at this vocabulary scale is challenging, and suffers from intolerable large model size and slow training speed, which leads to unsatisfying classification performance. A straightforward solution would be training separate expert networks (specialists), with each specialist focusing …
This paper develops a unified framework for image-to-image translation based on conditional diffusion models and evaluates this framework on four challenging image-to-image translation tasks, namely colorization, inpainting, uncropping, and JPEG restoration. Our simple implementation of image-to-image diffusion models outperforms strong GAN and regression baselines on all tasks, without task-specific hyper-parameter tuning, architecture customization, or any auxiliary loss or sophisticated new techniques needed. We uncover the impact of an L2 vs. L1 loss in the denoising diffusion objective on…
