DecompressionLM:从语言模型中确定性、诊断性地提取零样本概念图
DecompressionLM: Deterministic, Diagnostic, and Zero-Shot Concept Graph Extraction from Language Models
AI总结:
DecompressionLM通过无状态框架实现零样本概念图提取,揭示语言模型编码内容,解决跨序列耦合、竞争解码效应和可扩展性限制问题,提升压缩模型的知识广度和事实基础评估。
摘要:
现有知识探测方法依赖于预定义查询,限制了提取到已知概念的范围。我们引入DecompressionLM,一种无状态框架,用于零样本概念图提取,能够发现语言模型编码的内容,而无需预设查询或共享跨序列状态。我们的方法针对常见解码方法的三个限制:(i) 跨序列耦合将概率质量集中在高频前缀上,(ii) 竞争解码效应抑制长尾概念,(iii) 由于顺序探索产生的可扩展性限制。使用Van der Corput低分歧序列与算术解码,DecompressionLM实现了确定性、可并行生成,无需跨序列共享状态。在两个模型家族和五个量化变体中,我们发现激活感知量化(AWQ-4bit)将概念覆盖范围扩大30-170%,而均匀量化(GPTQ-Int4)导致71-86%的覆盖崩溃——这些行为无法可靠地通过解释层困惑度反映。基于语料库的验证进一步揭示了顶级与底部排名MMLU-Pro法律模型之间19.6点的幻觉差距。DecompressionLM将概念覆盖确立为评估压缩模型知识广度和事实基础的互补评价维度。
英文摘要:
Existing knowledge probing methods rely on pre-defined queries, limiting extraction to known concepts. We introduce DecompressionLM, a stateless framework for zero-shot concept graph extraction that discovers what language models encode without pre-specified queries or shared cross-sequence state. Our method targets three limitations of common decoding-based probing approaches: (i) cross-sequence coupling that concentrates probability mass on high-frequency prefixes, (ii) competitive decoding effects that suppress long-tail concepts, and (iii) scalability constraints arising from sequential exploration. Using Van der Corput low-discrepancy sequences with arithmetic decoding, DecompressionLM enables deterministic, embarrassingly parallel generation without shared state across sequences. Across two model families and five quantization variants, we find that activation-aware quantization (AWQ-4bit) expands concept coverage by 30-170%, while uniform quantization (GPTQ-Int4) induces 71-86% coverage collapse - divergent behaviors not reliably reflected by explanation-level perplexity. Corpus-based verification further reveals a 19.6-point hallucination gap between top- and bottom-ranked MMLU-Pro Law models. DecompressionLM establishes concept coverage as a complementary evaluation dimension for assessing knowledge breadth and factual grounding in compressed models intended for deployment. Code is available at https://github.com/ulab-uiuc/decompressionlm.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。