大语言模型的结构与冗余：通过随机矩阵理论的谱研究
Structure and Redundancy in Large Language Models: A Spectral Study via Random Matrix Theory
AI总结：
本研究通过谱几何和随机矩阵理论，提出EigenTrack用于检测大语言模型幻觉和分布外行为，以及RMT-KD用于压缩深度网络，提升模型可靠性与效率。
摘要：
本论文通过基于谱几何和随机矩阵理论（RMT）的统一框架，解决现代深度学习中持续存在且密切相关的问题，即可靠性与效率。随着深度网络和大语言模型的不断扩展，其内部行为变得越来越不透明，导致幻觉、在分布偏移下的脆弱泛化以及计算和能源需求的增加。通过分析隐藏激活在各层和输入上的特征值动态，本研究表明，谱统计提供了一个紧凑、稳定且可解释的模型行为观察窗口，能够将结构化的因果表示与噪声主导的变异性区分开来。在这一框架下，第一项贡献，EigenTrack，引入了一种实时方法，用于检测大语言和视觉-语言模型中的幻觉和分布外行为。EigenTrack将流式激活转换为谱描述符，如熵、方差和与Marchenko-Pastur基线的偏差，并利用轻量级的递归分类器建模其时间演变，从而在模型输出出现之前检测可靠性故障，同时提供对表示动态的可解释洞察。第二项贡献，RMT-KD，提出了一种通过随机矩阵理论知识蒸馏来压缩深度网络的系统方法。通过将激活谱中的异常特征值解释为任务相关的信息载体，RMT-KD通过迭代自我蒸馏将网络投影到较低维度的子空间中，从而产生显著更紧凑且能耗更低的模型，同时保持准确性和密集的硬件友好结构。
英文摘要：
This thesis addresses two persistent and closely related challenges in modern deep learning, reliability and efficiency, through a unified framework grounded in Spectral Geometry and Random Matrix Theory (RMT). As deep networks and large language models continue to scale, their internal behavior becomes increasingly opaque, leading to hallucinations, fragile generalization under distribution shift, and growing computational and energy demands. By analyzing the eigenvalue dynamics of hidden activations across layers and inputs, this work shows that spectral statistics provide a compact, stable, and interpretable lens on model behavior, capable of separating structured, causal representations from noise-dominated variability. Within this framework, the first contribution, EigenTrack, introduces a real-time method for detecting hallucinations and out-of-distribution behavior in large language and vision-language models. EigenTrack transforms streaming activations into spectral descriptors such as entropy, variance, and deviations from the Marchenko-Pastur baseline, and models their temporal evolution using lightweight recurrent classifiers, enabling early detection of reliability failures before they appear in model outputs while offering interpretable insight into representation dynamics. The second contribution, RMT-KD, presents a principled approach to compressing deep networks via random matrix theoretic knowledge distillation. By interpreting outlier eigenvalues in activation spectra as carriers of task-relevant information, RMT-KD progressively projects networks onto lower-dimensional subspaces through iterative self-distillation, yielding significantly more compact and energy-efficient models while preserving accuracy and dense, hardware-friendly structure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
