Building generalized models that can solve many computer vision tasks simultaneously is an intriguing direction. Recent works have shown image itself can be used as a natural interface for general-purpose visual perception and demonstrated inspiring results. In this paper, we explore diffusion-based vision generalists, where we unify different types of dense prediction tasks as conditional image generation and re-purpose pre-trained diffusion models for it. However, directly applying off-the-shelf latent diffusion models leads to a quantization issue. Thus, we propose to perform diffusion in pixel space and provide a recipe for finetuning pre-trained text-to-image diffusion models for dense vision tasks. In experiments, we evaluate our method on four different types of tasks and show competitive performance to the other vision generalists.
GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks
University of California, Santa Barbara（加州大学圣塔芭芭拉分校）；University of California, San Diego（加州大学圣迭戈分校）；University of Southern California（南加州大学）；Bytedance, US（字节跳动（美国））
Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Tasks
SenseTime Research（商汤科技研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；Xi’an Jiaotong University（西安交通大学）
Hidden-Shot: Towards One-Shot Task Generalization for Low-Level Vision Generalist Models
Pratt School of Engineering, Duke University（杜克大学普拉特工程学院）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）；Khoury College of Computer Science, Northeastern University（东北大学库里计算机科学学院）
OmniFashion: Towards Generalist Fashion Intelligence via Multi-Task Vision-Language Learning
National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University（国家多媒体软件工程技术研究中心、人工智能研究院、计算机科学学院、武汉大学）；Harbin Institute of Technology（哈尔滨工业大学）
GLID: Pre-training a Generalist Encoder-Decoder Vision Model
CUHK MMLab（香港中文大学多媒体实验室）；SenseTime Research（商汤科技研究院）；Shanghai AI Laboratory（上海人工智能实验室）；CPII under InnoHK（香港创新科技署下属CPII）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能国际研究院）
Masked AutoDecoder is Effective Multi-Task Vision Generalist
S-Lab, Nanyang Technological University（南洋理工大学S-Lab）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Sensetime Research（商汤科技研究院）；College of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）
VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks
OpenGVLab（开放通用视觉实验室）；Shanghai AI Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；SenseTime Research（商汤科技研究院）；Nanjing University（南京大学）；The University of Hong Kong（香港大学）；Beijing Institute of Technology（北京理工大学）；The Hong Kong University of Science and Technology（香港科技大学）；The Chinese University of Hong Kong（香港中文大学）
VisionLLM v2 提出端到端通用多模态大模型，通过 super link 机制连接 MLLM 与任务专用解码器，并在数百个视觉语言任务上联合训练，以共享参数实现目标定位、姿态估计、图像生成编辑等多样化任务，性能可比肩专用模型。
Super-Generalist: Towards Comprehensive and Accurate Medical Image Understanding via Generalist-Specialist Synergy
DAMO Academy, Alibaba Group（达摩院，阿里巴巴集团）；Hupan Lab（湖畔实验室）；Department of Radiology, Ningbo No. 2 Hospital（宁波市第二医院放射科）；Department of Radiology, Guangdong Provincial People’s Hospital（广东省人民医院放射科）；Department of Radiology, Shanghai Institution of Pancreatic Disease（上海胰腺疾病研究所放射科）；Department of Radiology, Shengjing Hospital of China Medical University（中国医科大学附属盛京医院放射科）；The First Affiliated Hospital, Zhejiang University School of Medicine（浙江大学医学院附属第一医院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Department of Automation, Tsinghua University（自动化系，清华大学）；Tencent HunyuanX（腾讯混元）；Beijing University of Posts and Telecommunications（北京邮电大学）；University of Science and Technology Beijing（北京科技大学）
GiT: Towards Generalist Vision Transformer through Universal Language Interface
Peking University（北京大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；ETH Zurich（苏黎世联邦理工学院）
Learning A Low-Level Vision Generalist via Visual Task Prompt
University of Macau（澳门大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shenzhen Institute of Advanced Technology, CAS（中国科学院深圳先进技术研究院）；Shanghai Jiao Tong University（上海交通大学）；The Hong Kong Polytechnic University（香港理工大学）；State Key Laboratory of Internet of Things for Smart City, University of Macau（澳门大学智能物联网状态重点实验室）；Shenzhen University of Advanced Technology（深圳先进技术研究院）
From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuning
Singapore Eye Research Institute, Singapore National Eye Centre（新加坡眼科研究所、新加坡国家眼科中心）；Duke-National University of Singapore Medical School（杜克-新加坡国立大学医学院）；Byers Eye Institute, Stanford University（斯坦福大学拜尔斯眼科研究所）
OmniMRI: A Unified Vision--Language Foundation Model for Generalist MRI Interpretation
Athinoula A. Martinos Center for Biomedical Imaging（阿提诺拉A.马丁诺斯生物医学成像中心）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；University Campus Bio-Medico of Rome（罗马生物医学大学校园）
Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding
College of Computer Science and Technology, Zhejiang University-University of Illinois Urbana-Champaign Institute（浙江大学计算机科学与技术学院）；Stomatology Hospital, School of Stomatology, Zhejiang University School of Medicine（浙江大学口腔医院）；Alibaba Inc（阿里巴巴集团）；College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机科学与电子工程学院）；Angelalign Technology Inc.（Angelalign技术有限公司）；CFAR & IHPC, Agency for Science, Technology and Research（CFAR与IHPC，新加坡科技研究局）；Department of Orthodontics, Shanghai Ninth People’s Hospital, College of Stomatology, Shanghai Jiao Tong University（上海第九人民医院正畸科，上海交通大学口腔医学院）
Towards Generalist Intelligence in Dentistry: Vision Foundation Models for Oral and Maxillofacial Radiology
Shanghai Jiao Tong University（上海交通大学）；Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Department of Oral Craniomaxillofacial（上海第九人民医院，上海交通大学医学院，口腔颅面医学部）；Shanghai Jiao Tong University, School of Computer Science（上海交通大学，计算机科学学院）；University of Electronic Science and Technology of China, School of Mechanical and Electrical Engineering（电子科技大学，机械与电子工程学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University, College of Stomatology（上海交通大学，口腔医学院）；National Center for Stomatology（国家口腔医学中心）；National Clinical Medical Research Center for Oral Diseases（国家口腔疾病临床医学研究中心）；Shanghai Key Laboratory of Stomatology（上海口腔医学重点实验室）；Shanghai Research Institute of Stomatology（上海口腔研究所）；Chinese Academy of Medical Science, Research Unit of Oral and Maxillofacial Regenerative Medicine（中国医学科学院，口腔颌面再生医学研究单元）
Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policy
Shanghai AI Lab（上海人工智能实验室）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；MMLab, The Chinese University of Hong Kong（香港中文大学MMLab）；Peking University（北京大学）；SenseTime Research（商汤科技研究院）；Tsinghua University（清华大学）；HKISI, CAS（中国科学院香港中文大学研究所）
Generalist versus Specialist Vision Foundation Models for Ocular Disease and Oculomics
Institute of Ophthalmology, University College London（伦敦大学学院眼科研究所）；Hawkes Institute, University College London（伦敦大学学院霍克斯研究所）；NIHR Biomedical Research Centre at Moorfields Eye Hospital NHS Foundation Trust（莫菲尔德眼科医院 NHS 基金会信托 NIHR 生物医学研究中心）；Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院）；Centre for Innovation and Precision Eye Health, Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院创新与精准眼健康中心）；Singapore Eye Research Institute, Singapore National Eye Centre（新加坡国家眼科中心新加坡眼科研究所）；Chinese University of Hong Kong（香港中文大学）；Shanghai Jiao Tong University（上海交通大学）；MOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院教育部人工智能重点实验室）；Stadtspital Zürich（苏黎世市立医院）；Spross Research Institute（斯普罗斯研究所）；Tsinghua Medicine, Tsinghua University（清华大学清华医学）；Beijing Visual Science and Translational Eye Research Institute, Beijing Tsinghua Changgung Hospital（北京清华长庚医院北京视觉科学与转化眼科研究所）；University College London（伦敦大学学院）
Modality-Specialized Synergizers for Interleaved Vision-Language Generalists
Virginia Tech（弗吉尼亚理工大学）；Intuit AI Research（Intuit AI研究院）；Meta AI；The Chinese University of Hong Kong（香港中文大学）；University of California, Davis（加州大学戴维斯分校）
VisionFM: a Multi-Modal Multi-Task Vision Foundation Model for Generalist Ophthalmic Artificial Intelligence
The Chinese University of Hong Kong（香港中文大学）；Beijing Tongren Eye Center（北京同仁眼科中心）；Beijing Tongren Hospital（北京同仁医院）；Capital Medical University（首都医科大学）；King’s College London（伦敦国王学院）；Intelligent Vision Plus Technology Co., Ltd.（智影加科技有限公司）；Peking University Health Science Center（北京大学医学部）；SightAI Technology Pte. Ltd.（SightAI科技私人有限公司）；Handan City Eye Hospital（邯郸市眼科医院）；First Affiliated Hospital of Xi’an Jiaotong University（西安交通大学第一附属医院）；Chengdu University of Traditional Chinese Medicine（成都中医药大学）；Eye Center of Xiangya Hospital, Central South University（中南大学湘雅医院眼科中心）；Shanxi Eye Hospital Affiliated to Shanxi Medical University（山西医科大学附属眼科医院）；Precision Robotics (Hong Kong) Co., Ltd.（精密机器人（香港）有限公司）；Imperial College London（帝国理工学院）；Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院）；Tsinghua University（清华大学）；Beijing Tsinghua Changgang Hospital（北京清华长庚医院）；Singapore Eye Research Institute（新加坡眼科研究所）；Singapore National Eye Centre（新加坡国家眼科中心）
Unifying Biomedical Vision-Language Expertise: Towards a Generalist Foundation Model via Multi-CLIP Knowledge Distillation
Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine（放射肿瘤科，Winship癌症研究所，埃默里大学医学院）；Department of Biomedical Engineering, College of Engineering, Georgia Institute of Technology（生物医学工程系，工程学院，佐治亚理工学院）；Department of Computer Science and Mathematics, Laney Graduate School, Emory University（计算机科学与数学系，兰尼研究生院，埃默里大学）；School of Electrical and Computer Engineering, College of Engineering, Georgia Institute of Technology（电气与计算机工程系，工程学院，佐治亚理工学院）
What Matters in Building Vision-Language-Action Models for Generalist Robots
Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；ByteDance Research（字节跳动研究院）；CASIA MAIS-NLPR；Shanghai Jiao Tong University（上海交通大学）；National University of Singapore（新加坡国立大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）
OneDiff: A Generalist Model for Image Difference Captioning
Institute of Automation, Chinese Academy of Sciences, China（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences, China（中国科学院大学人工智能学院）
GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）
