视觉-语言模型的最新进展通过视觉指令微调，已在各类任务中展现出显著的泛化能力。然而，预训练视觉编码器与大语言模型（LLMs）之间的衔接 gap 成为整个网络的瓶颈。为改善跨模态对齐效果，现有研究通常采用覆盖更多视觉任务的视觉指令数据来微调模型以完成问答任务，但这类数据获取成本高昂，且未充分挖掘图像中蕴含的丰富上下文信息。本文首次尝试利用视觉指令数据中被忽视的上下文，以自监督方式训练模型“学习”如何提出高质量问题。基于此，我们提出了名为 SQ-LLaVA（面向大型视觉-语言助手的自提问机制）的新型框架。SQ-LLaVA 能够在分析视觉线索与先验语言知识的同时，灵活生成有意义的图像相关问题，标志着其泛化视觉理解能力达到了更高水平。此外，在更高质量的指令数据上微调 SQ-LLaVA 后，其性能相较于传统视觉指令微调方法有所提升。这一提升凸显了自提问技术在各类场景下实现更深层、更细致视觉内容理解方面的有效性。
英文摘要：
Recent advances in vision-language models have shown notable generalization in broad tasks through visual instruction tuning. However, bridging the gap between the pre-trained vision encoder and the large language models (LLMs) becomes the whole network's bottleneck. To improve cross-modality alignment, existing works usually consider more visual instruction data covering a broader range of vision tasks to fine-tune the model for question-answering, which, however, is costly to obtain and has not thoroughly explored the rich contextual information contained in images. This paper first attempts to harness the overlooked context within visual instruction data, training the model to self-supervised "learning" how to ask high-quality questions. In this way, we introduce a novel framework named SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant. SQ-LLaVA exhibits proficiency in generating flexible and meaningful image-related questions while analyzing the visual clue and prior language knowledge, signifying an advanced level of generalized visual understanding. Moreover, fine-tuning SQ-LLaVA on higher-quality instruction data shows a performance improvement compared with traditional visual-instruction tuning methods. This improvement highlights the efficacy of self-questioning techniques in achieving a deeper and more nuanced comprehension of visual content across various contexts.
ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；School of Engineering, Westlake University（西湖大学工程学院）；University of Minnesota – Twin Cities（明尼苏达大学双城分校）；University of Toronto（多伦多大学）
PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understanding
Chongqing University of Posts and Telecommunications（重庆邮电大学）；Shanghai First Maternity and Infant Hospital（上海市第一妇婴保健院）；School of Medicine Tongji University（同济大学医学院）；College of Computer and Information Science, Chongqing Normal University（重庆师范大学计算机与信息科学学院）
LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound
ZJU-UIUC Institute, Zhejiang University（浙江大学ZJU-UIUC学院）；University of Washington（华盛顿大学）；Zhejiang University School of Medicine Sir Run Run Shaw Hospital（浙江大学医学院附属邵逸夫医院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
RetinalGPT: A Retinal Clinical Preference Conversational Assistant Powered by Large Vision-Language Models
School of Computing and Augmented Intelligence, Arizona State University（亚利桑那州立大学计算与增强智能学院）；School of Computing, Clemson University（克莱姆森大学计算机学院）；Washington University School of Medicine in St. Louis（圣路易斯华盛顿大学医学院）；The Keck School of Medicine of the University of Southern California（南加州大学凯克医学院）；Mayo Clinic（梅奥诊所）；Banner Alzheimer’s Institute（班纳阿尔茨海默病研究所）
Task-Oriented Semantic Communication in Large Multimodal Models-based Vehicle Networks
Institute of Science and Engineering, Kanazawa University（金泽大学科学与工程研究所）；Department of Electrical and Electronic Engineering, University of Hong Kong（香港大学电子与电气工程系）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
IC-EO: Interpretable Code-based assistant for Earth Observation
Université Paris Cité, LIPADE, F-75006 Paris, France askEarth AG, Zurich, Switzerland ESA -lab, Frascati, Italy ESA, Advanced Concepts；Studies Office, Noordwijk, the Netherlands
Agri-LLaVA: Knowledge-Infused Large Multimodal Assistant on Agricultural Pests and Diseases
China Three Gorges University（三峡大学）；College of Computer and Information Technology, China Three Gorges University（三峡大学计算机与信息学院）；University of Birmingham（伯明翰大学）；Southern University of Science and Technology（南方科技大学）
3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer
Australian Institute for Machine Learning, The University of Adelaide（澳大利亚机器学习研究所，阿德莱德大学）；Microsoft Research（微软研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）
本文提出 3D-LLaVA，一种仅以点云为输入的极简 3D 多模态大模型，通过集成全能超点 Transformer 实现视觉特征选择、提示编码与掩码解码，经混合预训练和指令微调，在多项基准上取得出色结果。
A Data Synthesis Method Driven by Large Language Models for Proactive Mining of Implicit User Intentions in Tourism
School of Computer & Communication Engineering, University of Science and Technology Beijing（信息与通信工程学院，北京科技大学）；School of Computer Science, University of South China（计算机科学学院，南方大学）；Department of Computer Science, Blekinge institute of Technology（计算机科学系，布莱京厄学院）
