VILA-U is a Unified foundation model that integrates Video, Image, Language understanding and generation. Traditional visual language models (VLMs) use separate modules for understanding and generating visual content, which can lead to misalignment and increased complexity. In contrast, VILA-U employs a single autoregressive next-token prediction framework for both tasks, eliminating the need for additional components like diffusion models. This approach not only simplifies the model but also achieves near state-of-the-art performance in visual language understanding and generation. The success of VILA-U is attributed to two main factors: the unified vision tower that aligns discrete visual tokens with textual inputs during pretraining, which enhances visual perception, and autoregressive image generation can achieve similar quality as diffusion models with high-quality dataset. This allows VILA-U to perform comparably to more complex models using a fully token-based autoregressive framework.
Integrating Pathology Foundation Models and Spatial Transcriptomics for Cellular Decomposition from Histology Images
School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USA（电气与计算机工程学院，佐治亚理工学院）；Department of Biomedical Engineering, Georgia Institute of Technology（生物医学工程系，佐治亚理工学院）；Emory University, Atlanta, GA, USA（埃默里大学）
This open access book provides a comprehensive overview of the state of the art in research and applications of Foundation Models and is intended for readers familiar with basic Natural Language Processing (NLP) concepts. Over the recent years, a revolutionary new paradigm has been developed for training models for NLP. These models are first pre-trained on large collections of text documents to acquire general syntactic knowledge and semantic information. Then, they are fine-tuned for specific tasks, which they can often solve with superhuman accuracy. When the models are large enough, they c…
Integrating SAM Supervision for 3D Weakly Supervised Point Cloud Segmentation
Department of Electrical and Computer Engineering, National University of Singapore（新加坡国立大学电子与计算机工程系）；SenseTime Research（商汤科技研究院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR)（信息与通信研究 institute（I2R），科技研究局（A*STAR））；School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学学院）
TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation
ARC Lab, Tencent PCG（腾讯PCG广告实验室）；City University of Hong Kong（香港城市大学）；Zhejiang University（浙江大学）；NLPR & MAIS, Institute of Automation, CAS, Beijing（自动化研究所）
Enhancing Representation in Medical Vision-Language Foundation Models via Multi-Scale Information Extraction Techniques
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Peng Cheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）；The University of Hong Kong（香港大学）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））
Text-guided Foundation Model Adaptation for Long-Tailed Medical Image Classification
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）；University of British Columbia（不列颠哥伦比亚大学）；Jiaxing Research Institute, Southern University of Science and Technology（南方科技大学嘉兴研究院）
Pedestrian Intention Prediction via Vision-Language Foundation Models
Institute for Transport Studies, Faculty of Environment, Computer Vision and Machine Learning Group, University of Leeds（运输研究学院、环境学院、计算机视觉与机器学习小组、莱斯特大学）；AI Centre, Department of Computer Science, University College London（人工智能中心、计算机科学系、伦敦大学学院）
Advancing Stroke Risk Prediction Using a Multi-modal Foundation Model
Signal Processing Institute EPFL University Lausanne（瑞士洛桑联邦理工学院信号处理研究所）；Brigham and Women’s Hospital Harvard Medical School（哈佛医学院布里奇沃特医院）；Department of Quantitative Biomedicine University of Zurich（苏黎世大学定量生物医学系）；ETH AI Center, Department of Computer Science Department of Quantitative Biomedicine University of Zurich（苏黎世大学定量生物医学系）
