Vision-language models have made significant strides recently, demonstrating superior performance across a range of tasks, e.g. optical character recognition and complex diagram analysis. Building on this trend, we introduce a new vision-language model, POINTS1.5, designed to excel in various real-world applications. POINTS1.5 is an enhancement of POINTS1.0 and incorporates several key innovations: i) We replace the original CLIP vision encoder, which had a fixed image resolution, with a NaViT-style vision encoder that supports native dynamic high resolution. This allows POINTS1.5 to process images of any resolution without needing to split them into tiles. ii) We add bilingual support to POINTS1.5, significantly enhancing its capability in Chinese. Due to the scarcity of open-source Chinese datasets for vision-language models, we collect numerous images from the Internet and annotate them using a combination of manual and automatic methods. iii) We propose a set of rigorous filtering methods for visual instruction tuning datasets. We comprehensively evaluate all these filtering methods, and choose the most effective ones to obtain the final visual instruction tuning set. Thanks to these innovations, POINTS1.5 significantly outperforms POINTS1.0 and demonstrates strong performance across a range of real-world applications. Notably, POINTS1.5-7B is trained on fewer than 4 billion tokens and ranks first on the OpenCompass leaderboard among models with fewer than 10 billion parameters
Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Model
School of Software Technology, Zhejiang University(浙江大学软件学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Electronic and Information, Shanghai Dianji University(上海电机学院电子信息学院)
VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework
Stanford University(斯坦福大学);Peking University(北京大学);Harvard University(哈佛大学);University of Washington, Seattle(华盛顿大学西雅图分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机与软件学院);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Research Center of Artificial Intelligence, Peng Cheng Laboratory(鹏城实验室人工智能研究中心)
LM4LV: A Frozen Large Language Model for Low-level Vision Tasks
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Nanjing University(南京大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Models
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所基础模型研究中心);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Peng Cheng Laboratory(鹏城实验室);Wuhan AI Research(武汉人工智能研究院)
VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks
OpenGVLab(开放通用视觉实验室);Shanghai AI Laboratory(上海人工智能实验室);Tsinghua University(清华大学);SenseTime Research(商汤科技研究院);Nanjing University(南京大学);The University of Hong Kong(香港大学);Beijing Institute of Technology(北京理工大学);The Hong Kong University of Science and Technology(香港科技大学);The Chinese University of Hong Kong(香港中文大学)
VisionLLM v2 提出端到端通用多模态大模型,通过 super link 机制连接 MLLM 与任务专用解码器,并在数百个视觉语言任务上联合训练,以共享参数实现目标定位、姿态估计、图像生成编辑等多样化任务,性能可比肩专用模型。
HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding
Tsinghua University(清华大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Johns Hopkins University(约翰斯·霍普金斯大学);SenseTime Research(商汤科技研究院);Nanjing University(南京大学)
From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models
School of Computer Science, Sichuan University(四川大学计算机学院);School of Electronic and Computer Engineering, Peking University Shenzhen Graduate School(北京大学深圳研究生院电子与计算机工程学院);Institute of Artificial Intelligence (TeleAI), China Telecom and Northwestern Polytechnical University(中国电信与西北工业大学人工智能研究院(TeleAI))
Towards Mitigating Modality Bias in Vision-Language Models for Temporal Action Localization
UVLab, Department of Computer Science, University of Warwick(华威大学计算机科学系UVLab);Department of Automation, University of Cambridge(剑桥大学自动化系);Department of Computer Science, The University of Sheffield(谢菲尔德大学计算机科学系)
A Pilot Study on Curator-Guided Multilingual Art Description for Blind and Low-Vision Audiences with Small Vision-Language Models
Technical University of Munich(慕尼黑技术大学);Foundation for Research and Technology -- Hellas(希腊研究与技术基金会);National University of Science and Technology Politehnica Bucharest(布加勒斯特政治技术科学与技术国家大学)