arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新，休息一下吧，周末愉快~~
arXiv 2606.00909cs.CLcs.AI

MLLM-Microscope：解锁多模态大语言模型中的隐藏结构

MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models

Satbayev University（萨特拜耶夫大学）

Ravil Mussabayev, Rustam Mussabayev

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出MLLM-Microscope系统，通过分析线性度、内在维度和各向异性，揭示多模态大语言模型中隐藏的表示结构，并基于ScienceQA数据集评估LLaVA-NeXT和OmniFusion，发现模态融合方式显著影响模型内部工作机理。

摘要：

本文提出MLLM-Microscope，一个用于分析多模态大语言模型（MLLMs）中隐藏表示的新型系统。我们的系统评估了跨transformer层的多模态token嵌入的线性度、内在维度和各向异性。利用ScienceQA数据集，我们评估了两个最先进的MLLM：LLaVA-NeXT和OmniFusion。我们发现，两种模态的token的主流和残差流在transformer层中均表现出高度线性行为。然而，LLaVA-NeXT的图像token线性度略有下降，而OmniFusion的保持一致。与LLaVA-NeXT相比，OmniFusion的图像token维度在各层中始终较高。此外，观察到OmniFusion的各向异性在各层中保持较低水平。这些发现表明，MLLM的内部工作高度依赖于将token序列传入LLM之前执行的模态融合的性质。这一发现以及从我们的系统中获得的其他潜在新见解，无疑能够增强我们对MLLM内部工作的理解，为未来的模型设计和优化提供信息。

英文摘要：

This work presents MLLM-Microscope, a novel system designed for analyzing the hidden representations within Multimodal Large Language Models (MLLMs). Our system evaluates the linearity, intrinsic dimension, and anisotropy of multimodal token embeddings across transformer layers. Utilizing the ScienceQA dataset, we evaluate two state-of-the-art MLLMs, LLaVA-NeXT and OmniFusion. We find that both the main and residual streams for tokens of both modalities exhibit highly linear behaviors across transformer layers. However, LLaVA-NeXT's image tokens reveal a slight decline in linearity, whereas OmniFusion's remain consistent. Image token dimensions in OmniFusion remain consistently higher across layers compared to LLaVA-NeXT. Also, the OmniFusion's anisotropy is observed to stay consistently low throughout the layers. These findings suggest that the inner workings of MLLMs highly depend on the nature of modality fusion performed before passing the token sequence into LLM. This and other new potential insights obtainable from our system are surely capable of enhancing our understanding of the inner workings of MLLMs, informing future model design and optimization.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑