arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.11729cs.AIcs.LGcs.SE
收藏

跨架构模型差异分析与Crosscoders:无监督发现不同LLM之间的差异

Cross-Architecture Model Diffing with Crosscoders: Unsupervised Discovery of Differences Between LLMs

Anthropic(Anthropic公司);Mila(米拉研究所);Université de Montréal(蒙特利尔大学)

Thomas Jiralerspong, Trenton Bricken

AI总结:

本文提出Dedicated Feature Crosscoders,通过无监督方法发现不同LLM之间的行为差异,如中国共产党一致性、美国例外论和版权拒绝机制。

摘要:

模型差异分析,即通过比较模型内部表示来识别其差异的过程,是揭示新模型中安全关键行为的一种有前途的方法。然而,其应用迄今为止主要集中在比较基础模型与其微调模型之间。由于新LLM发布往往采用新颖的架构,跨架构方法对于使模型差异分析广泛适用至关重要。Crosscoders是一种能够实现跨架构模型差异分析的解决方案,但迄今为止仅应用于基础模型与微调模型之间的比较。我们首次将Crosscoders应用于跨架构模型差异分析,并引入了专用特征Crosscoders(DFCs),这是一种架构修改,旨在更好地隔离某一模型独有的特征。利用这种技术,我们发现了一些特征,包括在Qwen3-8B和Deepseek-R1-0528-Qwen3-8B中的中国共产党一致性、在Llama3.1-8B-Instruct中的美国例外论,以及在GPT-OSS-20B中的版权拒绝机制。这些结果共同致力于确立跨架构Crosscoder模型差异分析作为识别AI模型之间有意义行为差异的有效方法。

英文摘要:

Model diffing, the process of comparing models' internal representations to identify their differences, is a promising approach for uncovering safety-critical behaviors in new models. However, its application has so far been primarily focused on comparing a base model with its finetune. Since new LLM releases are often novel architectures, cross-architecture methods are essential to make model diffing widely applicable. Crosscoders are one solution capable of cross-architecture model diffing but have only ever been applied to base vs finetune comparisons. We provide the first application of crosscoders to cross-architecture model diffing and introduce Dedicated Feature Crosscoders (DFCs), an architectural modification designed to better isolate features unique to one model. Using this technique, we find in an unsupervised fashion features including Chinese Communist Party alignment in Qwen3-8B and Deepseek-R1-0528-Qwen3-8B, American exceptionalism in Llama3.1-8B-Instruct, and a copyright refusal mechanism in GPT-OSS-20B. Together, our results work towards establishing cross-architecture crosscoder model diffing as an effective method for identifying meaningful behavioral differences between AI models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑