We train a suite of multimodal foundation models (MMFM) using the popular LLaVA framework with the recently released Gemma family of large language models (LLMs). Of particular interest is the 2B parameter Gemma model, which provides opportunities to construct capable small-scale MMFMs. In line with findings from other papers in this space, we test the effect of ablating three design features: pretraining the connector, utilizing a more powerful image backbone, and increasing the size of the language backbone. The resulting models, which we call LLaVA-Gemma, exhibit moderate performance on an array of evaluations, but fail to improve past the current comparably sized SOTA models. Closer analysis of performance shows mixed effects; skipping pretraining tends to reduce performance, larger vision models sometimes improve performance, and increasing language model size has inconsistent effects. We publicly release training recipes, code and weights for our models for the LLaVA-Gemma models.
HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Models
Simula Metropolitan Center for Digital Engineering (SimulaMet)（Simula数字工程研究中心）；Oslo Metropolitan University (OsloMet)（奥斯陆 Metropolitan 大学）；Simula Research Laboratory（Simula研究实验室）
Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decoding
Carnegie Mellon University（卡内基梅隆大学）；University of Nottingham（诺丁汉大学）；The University of Hong Kong（香港大学）；The Hong Kong Polytechnic University（香港理工大学）；Columbia University（哥伦比亚大学）；University of California, Berkeley（加州大学伯克利分校）
DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding
Courant Institute of Mathematical Sciences, New York University（纽约大学数学科学学院）；Tandon School of Engineering, New York University（纽约大学工程学院）；Cerebras Systems Inc.（Cerebras Systems公司）；University of Pennsylvania（宾夕法尼亚大学）
Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）；Institute of Artificial Intelligence, Xiamen University（厦门大学人工智能研究院）
Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（多模态人工智能系统国家重点实验室，中国科学院自动化所）；School of Artificial Intelligence, UCAS（人工智能学院，中国科学院大学）；Beijing National Research Center for Information Science and Technology（北京信息科学研究中心）；Institute of Artificial Intelligence, USTB（信息科学技术大学人工智能学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Zhongguancun Academy（中关村学院）
