arXiv 2510.232642025-10-27cs.LGcs.AI

PAHQ：通过混合精度推理优化加速自动电路发现

PAHQ: Accelerating Automated Circuit Discovery through Mixed-Precision Inference Optimization

King Abdullah University of Science and Technology（卡布斯大学）；Harbin Institute of Technology（哈尔滨工业大学）

Xinhai Wang, Shu Yang, Liangyu Wang, Lin Zhang, Huanyi Xie, Lijie Hu, Di Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出PAHQ方法，通过混合精度量化优化激活修补操作，加速自动电路发现，减少高达80%运行时间和30%内存，同时保持分析忠实性，并可与现有技术集成。

摘要：

电路发现涉及识别预训练语言模型中稀疏且与任务相关的子网络，是机制可解释性的基石。自动电路发现（ACDC）已成为电路发现中的关键方法，但其在大型语言模型上的应用受到计算效率低下和内存需求过高的严重限制。尽管已提出多种加速方法，但它们主要依赖于对ACDC的线性近似，这显著损害了分析的忠实性。我们提出的加速自动电路发现方法——逐注意力头量化（PAHQ）——采取了一种根本不同的方法，通过优化每个单独修补操作的效率来加速。PAHQ利用了激活修补与混合精度量化（MPQ）之间的基本对齐：通过修补进行的可解释性分析本质上执行了针对性的消融研究。因此，我们可以仅对调查的组件保持高精度，同时安全地降低网络中其他部分的精度。与未加速的ACDC相比，PAHQ加速的ACDC将运行时间减少了高达80%，内存消耗减少了高达30%，同时保持了忠实性。重要的是，我们的方法通过修改注意力计算机制，可以轻松与现有的基于边的电路发现技术集成。这种免训练的方法为加速机制可解释性方法提供了一条实用且新颖的途径。我们的代码可在https://github.com/626619403/PAHQ获取。

英文摘要：

Circuit discovery, which involves identifying sparse and task-relevant subnetworks in pre-trained language models, is a cornerstone of mechanistic interpretability. Automated Circuit Discovery (ACDC) has emerged as a pivotal methodology in circuit discovery, but its application to large language models is severely limited by computational inefficiency and prohibitively high memory requirements. Although several accelerated approaches have been proposed, they primarily rely on linear approximations to ACDC, which significantly compromises analytical faithfulness. Our proposed method for accelerating automated circuit discovery, Per Attention Head Quantization (PAHQ), takes a fundamentally different approach by optimizing the efficiency of each individual patching operation. PAHQ leverages a fundamental alignment between activation patching and mixed-precision quantization (MPQ): interpretability analysis through patching essentially performs targeted ablation studies. Therefore, we can maintain high precision exclusively for investigated components while safely reducing precision elsewhere in the network. PAHQ-accelerated ACDC reduces runtime by up to 80\% and memory consumption by up to 30\% compared to unaccelerated ACDC while maintaining faithfulness. Importantly, our method readily integrates with existing edge-based circuit discovery techniques by modifying the attention computation mechanism. This training-free approach provides a practical and novel pathway for accelerating mechanistic interpretability methods. Our code is available at https://github.com/626619403/PAHQ.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。