GiT: Towards Generalist Vision Transformer through Universal Language Interface
Peking University（北京大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；ETH Zurich（苏黎世联邦理工学院）
Haiyang Wang, Hao Tang, Li Jiang, Shaoshuai Shi, Muhammad Ferjad Naeem, Hongsheng Li, Bernt Schiele, Liwei Wang
This paper proposes a simple, yet effective framework, called GiT, simultaneously applicable for various vision tasks only with a vanilla ViT. Motivated by the universality of the Multi-layer Transformer architecture (e.g, GPT) widely used in large language models (LLMs), we seek to broaden its scope to serve as a powerful vision foundation model (VFM). However, unlike language modeling, visual tasks typically require specific modules, such as bounding box heads for detection and pixel decoders for segmentation, greatly hindering the application of powerful multi-layer transformers in the vision domain. To solve this, we design a universal language interface that empowers the successful auto-regressive decoding to adeptly unify various visual tasks, from image-level understanding (e.g., captioning), over sparse perception (e.g., detection), to dense prediction (e.g., segmentation). Based on the above designs, the entire model is composed solely of a ViT, without any specific additions, offering a remarkable architectural simplification. GiT is a multi-task visual model, jointly trained across five representative benchmarks without task-specific fine-tuning. Interestingly, our GiT builds a new benchmark in generalist performance, and fosters mutual enhancement across tasks, leading to significant improvements compared to isolated training. This reflects a similar impact observed in LLMs. Further enriching training with 27 datasets, GiT achieves strong zero-shot results over various tasks. Due to its simple design, this paradigm holds promise for narrowing the architectural gap between vision and language. Code and models will be available at \url{https://github.com/Haiyang-W/GiT}.
3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer
Australian Institute for Machine Learning, The University of Adelaide（澳大利亚机器学习研究所，阿德莱德大学）；Microsoft Research（微软研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）
本文提出 3D-LLaVA，一种仅以点云为输入的极简 3D 多模态大模型，通过集成全能超点 Transformer 实现视觉特征选择、提示编码与掩码解码，经混合预训练和指令微调，在多项基准上取得出色结果。
Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policy
Shanghai AI Lab（上海人工智能实验室）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；MMLab, The Chinese University of Hong Kong（香港中文大学MMLab）；Peking University（北京大学）；SenseTime Research（商汤科技研究院）；Tsinghua University（清华大学）；HKISI, CAS（中国科学院香港中文大学研究所）
Masked AutoDecoder is Effective Multi-Task Vision Generalist
S-Lab, Nanyang Technological University（南洋理工大学S-Lab）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Sensetime Research（商汤科技研究院）；College of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）
Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Tasks
SenseTime Research（商汤科技研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；Xi’an Jiaotong University（西安交通大学）
MGA: Medical generalist agent through text-guided knowledge transformation
Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institutes of Advanced Technology（中国科学院深圳先进技术研究院保罗·C·劳特伯生物医学成像研究中心）；University of Chinese Academy of Sciences（中国科学院大学）；Pengcheng Laboratory（鹏城实验室）；Renmin Hospital of Wuhan University（武汉大学人民医院）
Learning A Low-Level Vision Generalist via Visual Task Prompt
University of Macau（澳门大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shenzhen Institute of Advanced Technology, CAS（中国科学院深圳先进技术研究院）；Shanghai Jiao Tong University（上海交通大学）；The Hong Kong Polytechnic University（香港理工大学）；State Key Laboratory of Internet of Things for Smart City, University of Macau（澳门大学智能物联网状态重点实验室）；Shenzhen University of Advanced Technology（深圳先进技术研究院）
From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuning
Singapore Eye Research Institute, Singapore National Eye Centre（新加坡眼科研究所、新加坡国家眼科中心）；Duke-National University of Singapore Medical School（杜克-新加坡国立大学医学院）；Byers Eye Institute, Stanford University（斯坦福大学拜尔斯眼科研究所）
Mora: Enabling Generalist Video Generation via A Multi-Agent Framework
University of Notre Dame（圣母大学）；Lehigh University（理海大学）；LinkedIn Corporation（领英公司）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；Nanyang Technological University（南洋理工大学）；Peking University（北京大学）；Microsoft Research（微软研究院）
Towards Generalist Intelligence in Dentistry: Vision Foundation Models for Oral and Maxillofacial Radiology
Shanghai Jiao Tong University（上海交通大学）；Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Department of Oral Craniomaxillofacial（上海第九人民医院，上海交通大学医学院，口腔颅面医学部）；Shanghai Jiao Tong University, School of Computer Science（上海交通大学，计算机科学学院）；University of Electronic Science and Technology of China, School of Mechanical and Electrical Engineering（电子科技大学，机械与电子工程学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University, College of Stomatology（上海交通大学，口腔医学院）；National Center for Stomatology（国家口腔医学中心）；National Clinical Medical Research Center for Oral Diseases（国家口腔疾病临床医学研究中心）；Shanghai Key Laboratory of Stomatology（上海口腔医学重点实验室）；Shanghai Research Institute of Stomatology（上海口腔研究所）；Chinese Academy of Medical Science, Research Unit of Oral and Maxillofacial Regenerative Medicine（中国医学科学院，口腔颌面再生医学研究单元）
GIT: A Generative Image-to-text Transformer for Vision and Language
Microsoft Cloud and AI（微软云与人工智能部门）
在本文中，我们设计和训练一个生成的图像到文本的Transformer，GIT，统一的视觉语言任务，如图像/视频描述和问答。虽然生成模型在预训练和微调之间提供了一致的网络架构，但现有的工作通常包含复杂的结构（单/多模态编码器/解码器），并依赖于外部模块，如对象检测器/标签和光学字符识别（OCR）。在GIT中，我们将架构简化为单一语言建模任务下的一个图像编码器和一个文本解码器。我们还扩展了预训练数据和模型大小，以提高模型性能。没有花里胡哨，我们的GIT在12个具有挑战性的基准上建立了新的艺术状态。例如，我们的模型在TextCaps上首次超过了人类的表现（138.2 vs. CIDEr中的125.5）。此外，我们提出了一种新的方案，基于生成的图像分类和场景文本识别，在标准基准测试中取得了不错的性能。代码在\url{https://github.com/microsoft/GenerativeImage2Text}发布。
Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding
College of Computer Science and Technology, Zhejiang University-University of Illinois Urbana-Champaign Institute（浙江大学计算机科学与技术学院）；Stomatology Hospital, School of Stomatology, Zhejiang University School of Medicine（浙江大学口腔医院）；Alibaba Inc（阿里巴巴集团）；College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机科学与电子工程学院）；Angelalign Technology Inc.（Angelalign技术有限公司）；CFAR & IHPC, Agency for Science, Technology and Research（CFAR与IHPC，新加坡科技研究局）；Department of Orthodontics, Shanghai Ninth People’s Hospital, College of Stomatology, Shanghai Jiao Tong University（上海第九人民医院正畸科，上海交通大学口腔医学院）
Brain3D: Brain Report Automation via Inflated Vision Transformers in 3D
University of Naples Federico II, Department of Electrical Engineering and Information Technology（那不勒斯费德里科二世大学电子工程与信息技术系）；Northwestern University, Dept. of Computer Science, McCormick School of Engineering and Applied Science（西北大学计算机科学系，工程与应用科学学院）；Pegaso University, Department of Information Science and Technology（佩加索大学信息科学与技术系）
South China University of Technology（南方科技大学）；INTSIG Information Co. Ltd（INTSIG信息有限公司）；INTSIG-SCUT Joint Lab of Document Image Analysis（INTSIG-SCUT文档图像分析联合实验室）
What Matters in Building Vision-Language-Action Models for Generalist Robots
Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；ByteDance Research（字节跳动研究院）；CASIA MAIS-NLPR；Shanghai Jiao Tong University（上海交通大学）；National University of Singapore（新加坡国立大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）
