arXiv 2606.075242026-09-01cs.CLcs.AI

ABLE：基于归因的大模型嵌入表示与映射

ABLE: Representing and Mapping LLMs via Attribution-Based Large-model Embedding

The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Deep Interdisciplinary Intelligence Lab (DI2 Lab)（深度跨学科智能实验室（DI2 Lab））

Zirui Wang, Yusen Hou, Shaofeng Liang, Bowen Tian, Yanlin Zhang, Wenshuo Chen, Yutao Yue

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出ABLE框架，利用梯度特征归因和分词器无关的词级对齐构建模型嵌入，实现异构LLM的高效比较，在关系预测、模型路由和基准分数预测上表现优异。

摘要：

大语言模型（LLM）的爆炸式增长形成了一个异构且文档不完善的生态系统，使得系统性的模型比较对于来源审计、安全分析和模型选择越来越重要。现有的表示方法难以高效应对这一场景。分析内部参数的方法在架构兼容时很强大，但在结构异构下面临可扩展性障碍；而依赖外部输出的方法可能混淆具有相似行为的模型，且难以在不同分词器的更丰富输出空间中对齐。为弥合这一差距，我们提出ABLE（基于归因的大模型嵌入）框架，利用可解释性空间构建模型表示。通过基于梯度的特征归因，经由分词器无关的词级对齐进行聚合，ABLE捕获模型特定的输入敏感性模式，而不仅仅是表面输出。除经验效用外，我们提供了稳定性分析，表明在可微Transformer风格模型的标准正则性假设下，ABLE诱导出一个Lipschitz连续的参数到嵌入映射，并具有有限样本收敛保证。在239个开源LLM上的大量实验表明，我们的无训练方法在关系预测、模型路由和基准分数预测方面达到了有竞争力或更优的性能。

英文摘要：

The explosive growth of large language models (LLMs) has created a heterogeneous and poorly documented ecosystem, making systematic model comparison increasingly important for provenance auditing, security analysis, and model selection. Existing representation methods struggle to address this setting efficiently. Approaches analyzing internal parameters are powerful when architectures are compatible, but face scalability barriers under structural heterogeneity, while methods relying on external outputs may conflate models with similar behaviors and are difficult to align in richer output spaces across different tokenizers. To bridge this gap, we propose ABLE (Attribution-Based Large-model Embedding), a framework that leverages the interpretability space to construct model representations. By aggregating gradient-based feature attributions via a tokenizer-agnostic word-level alignment, ABLE captures model-specific input-sensitivity patterns rather than only surface-level outputs. Beyond empirical utility, we provide a stability analysis showing that, under standard regularity assumptions for differentiable Transformer-style models, ABLE induces a Lipschitz-continuous parameter-to-embedding map with finite-sample convergence guarantees. Extensive experiments on 239 open-source LLMs demonstrate that our training-free approach achieves competitive or superior performance in relation prediction, model routing, and benchmark score prediction.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。