arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.00716cs.AIcs.LG
收藏

CircuitProbe: 通过稳定性区检测预测Transformer中的推理电路

CircuitProbe: Predicting Reasoning Circuits in Transformers via Stability Zone Detection

Independent Researcher(独立研究员)

Rajkiran Panuganti

AI总结:

CircuitProbe通过激活统计预测Transformer中的推理电路,利用稳定性区和幅度区检测,提升推理效率,验证了其在多种模型上的有效性。

摘要:

Transformer语言模型包含局部化的推理电路,连续的层块在推理时复制可提高推理能力。目前寻找这些电路需要暴力扫描,每模型消耗25个GPU小时。我们提出CircuitProbe,通过激活统计在CPU上仅需5分钟即可预测电路位置,速度提升三到四个数量级。我们发现推理电路分为两种类型:早期层的稳定性电路,通过表征变化的导数检测;晚期层的幅度电路,通过异常评分检测。我们在9个模型上验证,涵盖6种架构,包括2025模型,确认CircuitProbe的预测与最优电路匹配或相差2层以内。在Qwen 2.5系列的扩展实验中发现,层复制在3B参数以下模型中有益,但7B+模型中性能下降,使其成为小型语言模型的实用扩展技术。CircuitProbe仅需10个校准示例,其预测在英语、印地语、中文和法语中均稳定。

英文摘要:

Transformer language models contain localized reasoning circuits, contiguous layer blocks that improve reasoning when duplicated at inference time. Finding these circuits currently requires brute-force sweeps costing 25 GPU hours per model. We propose CircuitProbe, which predicts circuit locations from activation statistics in under 5 minutes on CPU, providing a speedup of three to four orders of magnitude. We find that reasoning circuits come in two types: stability circuits in early layers, detected through the derivative of representation change, and magnitude circuits in late layers, detected through anomaly scoring. We validate across 9 models spanning 6 architectures, including 2025 models, confirming that CircuitProbe top predictions match or are within 2 layers of the optimal circuit in all validated cases. A scaling experiment across the Qwen 2.5 family reveals that layer duplication consistently benefits models under 3B parameters but degrades performance in 7B+ models, making this a practical scaling technique for small language models. CircuitProbe requires as few as 10 calibration examples and its predictions are stable across English, Hindi, Chinese, and French.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑