LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understanding
School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机科学与信息工程学院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Meituan(美团)
Hongyu Li, Jinyu Chen, Ziyu Wei, Shaofei Huang, Tianrui Hui, Jialin Gao, Xiaoming Wei, Si Liu
Recent advancements in multimodal large language models (MLLMs) have shown promising results, yet existing approaches struggle to effectively handle both temporal and spatial localization simultaneously. This challenge stems from two key issues: first, incorporating spatial-temporal localization introduces a vast number of coordinate combinations, complicating the alignment of linguistic and visual coordinate representations; second, encoding fine-grained temporal and spatial information during video feature compression is inherently difficult. To address these issues, we propose LLaVA-ST, a MLLM for fine-grained spatial-temporal multimodal understanding. In LLaVA-ST, we propose Language-Aligned Positional Embedding, which embeds the textual coordinate special token into the visual space, simplifying the alignment of fine-grained spatial-temporal correspondences. Additionally, we design the Spatial-Temporal Packer, which decouples the feature compression of temporal and spatial resolutions into two distinct point-to-region attention processing streams. Furthermore, we propose ST-Align dataset with 4.3M training samples for fine-grained spatial-temporal multimodal understanding. With ST-align, we present a progressive training pipeline that aligns the visual and textual feature through sequential coarse-to-fine stages.Additionally, we introduce an ST-Align benchmark to evaluate spatial-temporal interleaved fine-grained understanding tasks, which include Spatial-Temporal Video Grounding (STVG) , Event Localization and Captioning (ELC) and Spatial Video Grounding (SVG). LLaVA-ST achieves outstanding performance on 11 benchmarks requiring fine-grained temporal, spatial, or spatial-temporal interleaving multimodal understanding. Our code, data and benchmark will be released at Our code, data and benchmark will be released at https://github.com/appletea233/LLaVA-ST .
Agri-LLaVA: Knowledge-Infused Large Multimodal Assistant on Agricultural Pests and Diseases
China Three Gorges University(三峡大学);College of Computer and Information Technology, China Three Gorges University(三峡大学计算机与信息学院);University of Birmingham(伯明翰大学);Southern University of Science and Technology(南方科技大学)
Understanding and Mitigating Toxicity in Image-Text Pretraining Datasets: A Case Study on LLaVA
Cohere for AI Community(Cohere for AI社区);Cisco Meraki;Indiana University Bloomington(印第安纳大学布卢明顿分校);Bangladesh University of Engineering and Technology(孟加拉工程与技术大学)
School of Artificial Intelligence, UCAS(人工智能学院,UCAS);State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(多模态人工智能系统国家重点实验室,CASIA);Centre for Artificial Intelligence and Robotics, HKISI-CAS(人工智能与机器人中心,HKISI-CAS);Huawei Inc(华为公司)
LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation
Alibaba(阿里巴巴);The Chinese University of Hong Kong(香港中文大学);Zhejiang University(浙江大学);Centre for Perceptual and Interactive Intelligence, Hong Kong(香港感知与交互智能中心);Beihang University(北京航空航天大学)
ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究所);School of Engineering, Westlake University(西湖大学工程学院);University of Minnesota – Twin Cities(明尼苏达大学双城分校);University of Toronto(多伦多大学)
FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models
Beijing Institute of Technology(北京理工大学);BIGAI(北京智源人工智能研究院);Peking University(北京大学);Monash University(莫纳什大学);Shenzhen MSU-BIT University(深圳北理莫斯科大学);Tsinghua University(清华大学)
LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Models
University of Electronic Science and Technology of China(电子科技大学);Chengdu University of Information Technology(成都信息工程大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳))
Hyper-LLaVA: Hyperbolic Uncertainty-aware Modality-Balanced Routing for Multimodal Continual Instruction Tuning
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机技术研究所);Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学)
MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment
CUHK MMLab(香港中文大学多媒体实验室);Waseda University(早稻田大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);SenseTime Research(商汤科技研究院);Shanghai AI Laboratory(上海人工智能实验室);CPII under InnoHK(创新香港旗下的CPII机构)
EmoMeta: A Multimodal Dataset for Fine-grained Emotion Classification in Chinese Metaphors
School of Software(软件学院);Dalian University of Technology(大连理工大学);School of Foreign Languages and School of Software(外语学院和软件学院);Faculty of Business Administration(商学院);University of Macau(澳门大学);School of Computing Technologies(计算技术学院);RMIT University(皇家墨尔本理工大学)
LLaVA-RadZ: Can Multimodal Large Language Models Effectively Tackle Zero-shot Radiology Recognition?
East China Normal University(华东师范大学);The Chinese University of Hong Kong(香港中文大学);Northwest A&F University(西北农林科技大学);Tencent Youtu Lab(腾讯优图实验室);Xiamen University(厦门大学)
LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Chinese Academy of Sciences(中国科学院);University of Chinese Academy of Sciences(中国科学院大学)
WSI-LLaVA: A Multimodal Large Language Model for Whole Slide Image
Shenzhen University(深圳大学);University of Nottingham Ningbo China(诺丁汉大学宁波分校);City University of Hong Kong(香港城市大学);Stanford University(斯坦福大学);Hong Kong University of Science and Technology(香港科学与技术大学)
AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularity
School of Informatics, Xiamen University, China(厦门大学信息学院);Pattern Recognition Center, WeChat AI, Tencent Inc, China(腾讯人工智能研究院);Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China(福建省和台湾非物质文化遗产数字化保护与智能处理重点实验室);Shanghai Artificial Intelligence Laboratory, China(上海人工智能实验室)