Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Tasks
SenseTime Research（商汤科技研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；Xi’an Jiaotong University（西安交通大学）
Despite the remarkable success of foundation models, their task-specific fine-tuning paradigm makes them inconsistent with the goal of general perception modeling. The key to eliminating this inconsistency is to use generalist models for general task modeling. However, existing attempts at generalist models are inadequate in both versatility and performance. In this paper, we propose Uni-Perceiver v2, which is the first generalist model capable of handling major large-scale vision and vision-language tasks with competitive performance. Specifically, images are encoded as general region proposals, while texts are encoded via a Transformer-based language model. The encoded representations are transformed by a task-agnostic decoder. Different tasks are formulated as a unified maximum likelihood estimation problem. We further propose an improved optimizer to ensure stable multi-task learning with an unmixed sampling strategy, which is helpful for tasks requiring large batch-size training. After being jointly trained on various tasks, Uni-Perceiver v2 is capable of directly handling downstream tasks without any task-specific adaptation. Results show that Uni-Perceiver v2 outperforms all existing generalist models in both versatility and performance. Meanwhile, compared with the commonly-recognized strong baselines that require tasks-specific fine-tuning, Uni-Perceiver v2 achieves competitive performance on a broad range of vision and vision-language tasks.
OmniFashion: Towards Generalist Fashion Intelligence via Multi-Task Vision-Language Learning
National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University（国家多媒体软件工程技术研究中心、人工智能研究院、计算机科学学院、武汉大学）；Harbin Institute of Technology（哈尔滨工业大学）
Super-Generalist: Towards Comprehensive and Accurate Medical Image Understanding via Generalist-Specialist Synergy
DAMO Academy, Alibaba Group（达摩院，阿里巴巴集团）；Hupan Lab（湖畔实验室）；Department of Radiology, Ningbo No. 2 Hospital（宁波市第二医院放射科）；Department of Radiology, Guangdong Provincial People’s Hospital（广东省人民医院放射科）；Department of Radiology, Shanghai Institution of Pancreatic Disease（上海胰腺疾病研究所放射科）；Department of Radiology, Shengjing Hospital of China Medical University（中国医科大学附属盛京医院放射科）；The First Affiliated Hospital, Zhejiang University School of Medicine（浙江大学医学院附属第一医院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Masked AutoDecoder is Effective Multi-Task Vision Generalist
S-Lab, Nanyang Technological University（南洋理工大学S-Lab）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Sensetime Research（商汤科技研究院）；College of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）
GiT: Towards Generalist Vision Transformer through Universal Language Interface
Peking University（北京大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；ETH Zurich（苏黎世联邦理工学院）
AnomalyMoE: Towards a Language-free Generalist Model for Unified Visual Anomaly Detection
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China（中国科学院大学人工智能学院）；Objecteye Inc., Beijing, China（Objecteye公司）
BRIGHT: A Collaborative Generalist-Specialist Foundation Model for Breast Pathology
Department of Breast Pathology and Laboratory, Tianjin Medical University Cancer Institute & Hospital, National Clinical Research Center for Cancer, Key Laboratory of Breast Cancer Prevention and Therapy, Tianjin Medical University, Ministry of Education, Tianjin’s Clinical Research Center for Cancer, West Huanhu Road, Tianjin, China（天津医科大学肿瘤医院乳腺病理科及实验室，国家癌症临床研究中心，天津医科大学乳腺癌预防与治疗重点实验室，天津医科大学，教育部，天津癌症临床研究中心，西湖南路，天津，中国）；Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application, Guangdong Provincial People’s Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, China（广东省人工智能在医学影像分析与应用重点实验室，广东省人民医院（广东省医学科学院），南方医科大学，广州，中国）
Learning A Low-Level Vision Generalist via Visual Task Prompt
University of Macau（澳门大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shenzhen Institute of Advanced Technology, CAS（中国科学院深圳先进技术研究院）；Shanghai Jiao Tong University（上海交通大学）；The Hong Kong Polytechnic University（香港理工大学）；State Key Laboratory of Internet of Things for Smart City, University of Macau（澳门大学智能物联网状态重点实验室）；Shenzhen University of Advanced Technology（深圳先进技术研究院）
VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks
OpenGVLab（开放通用视觉实验室）；Shanghai AI Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；SenseTime Research（商汤科技研究院）；Nanjing University（南京大学）；The University of Hong Kong（香港大学）；Beijing Institute of Technology（北京理工大学）；The Hong Kong University of Science and Technology（香港科技大学）；The Chinese University of Hong Kong（香港中文大学）
VisionLLM v2 提出端到端通用多模态大模型，通过 super link 机制连接 MLLM 与任务专用解码器，并在数百个视觉语言任务上联合训练，以共享参数实现目标定位、姿态估计、图像生成编辑等多样化任务，性能可比肩专用模型。
Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding
College of Computer Science and Technology, Zhejiang University-University of Illinois Urbana-Champaign Institute（浙江大学计算机科学与技术学院）；Stomatology Hospital, School of Stomatology, Zhejiang University School of Medicine（浙江大学口腔医院）；Alibaba Inc（阿里巴巴集团）；College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机科学与电子工程学院）；Angelalign Technology Inc.（Angelalign技术有限公司）；CFAR & IHPC, Agency for Science, Technology and Research（CFAR与IHPC，新加坡科技研究局）；Department of Orthodontics, Shanghai Ninth People’s Hospital, College of Stomatology, Shanghai Jiao Tong University（上海第九人民医院正畸科，上海交通大学口腔医学院）
OmniMRI: A Unified Vision--Language Foundation Model for Generalist MRI Interpretation
Athinoula A. Martinos Center for Biomedical Imaging（阿提诺拉A.马丁诺斯生物医学成像中心）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；University Campus Bio-Medico of Rome（罗马生物医学大学校园）
Department of Automation, Tsinghua University（自动化系，清华大学）；Tencent HunyuanX（腾讯混元）；Beijing University of Posts and Telecommunications（北京邮电大学）；University of Science and Technology Beijing（北京科技大学）
Unifying Biomedical Vision-Language Expertise: Towards a Generalist Foundation Model via Multi-CLIP Knowledge Distillation
Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine（放射肿瘤科，Winship癌症研究所，埃默里大学医学院）；Department of Biomedical Engineering, College of Engineering, Georgia Institute of Technology（生物医学工程系，工程学院，佐治亚理工学院）；Department of Computer Science and Mathematics, Laney Graduate School, Emory University（计算机科学与数学系，兰尼研究生院，埃默里大学）；School of Electrical and Computer Engineering, College of Engineering, Georgia Institute of Technology（电气与计算机工程系，工程学院，佐治亚理工学院）
GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks
University of California, Santa Barbara（加州大学圣塔芭芭拉分校）；University of California, San Diego（加州大学圣迭戈分校）；University of Southern California（南加州大学）；Bytedance, US（字节跳动（美国））
Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography
Department of Quantitative Biomedicine, University of Zurich（苏黎世大学定量生物医学系）；ETH AI Center, ETH Zurich（苏黎世联邦理工学院人工智能中心）；International School of Medicine, Istanbul Medipol University（伊斯坦布尔梅迪波尔大学国际医学院）；Department of Electrical and Computer Engineering, Boston University（波士顿大学电气与计算机工程系）；Division of Intramural Research, National Institutes of Health（美国国立卫生研究院内部研究部）；Department of Computing, Imperial College London（伦敦帝国理工学院计算机系）；Department of Computer Science, ETH Zurich（苏黎世联邦理工学院计算机科学系）；Institute for Diagnostic and Interventional Radiology, University Hospital Zurich（苏黎世大学医院诊断与介入放射学研究所）；Department Artificial Intelligence in Biomedical Engineering, FAU Erlangen-Nürnberg（埃尔朗根-纽伦堡工业大学生物医学工程人工智能系）
MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation
Centre for Perceptual and Interactive Intelligence, the Chinese University of Hong Kong（香港中文大学感知与交互智能中心）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；Department of Electronic Engineering, the Chinese University of Hong Kong（香港中文大学电子工程系）
Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policy
Shanghai AI Lab（上海人工智能实验室）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；MMLab, The Chinese University of Hong Kong（香港中文大学MMLab）；Peking University（北京大学）；SenseTime Research（商汤科技研究院）；Tsinghua University（清华大学）；HKISI, CAS（中国科学院香港中文大学研究所）
Modality-Specialized Synergizers for Interleaved Vision-Language Generalists
Virginia Tech（弗吉尼亚理工大学）；Intuit AI Research（Intuit AI研究院）；Meta AI；The Chinese University of Hong Kong（香港中文大学）；University of California, Davis（加州大学戴维斯分校）
Rethinking Facial Expression Recognition in the Era of Multimodal Large Language Models: Benchmark, Datasets, and Beyond
The Chinese University of Hong Kong（香港中文大学）；Peking University（北京大学）；Tencent（腾讯）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Tsinghua University（清华大学）；Nanyang Technological University（南洋理工大学）；Westlake University（西湖大学）
UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interface
Center for Data Science, Peking University（北京大学数据科学中心）；Alibaba Group（阿里巴巴集团）；CASIA；Center for Machine Learning Research, Peking University（北京大学机器学习研究中心）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）
GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）
METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Model
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机学院，北京大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
