arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.11852cs.AIcs.CLcs.LG
收藏

原型Transformer:迈向可解释设计的语言模型架构

Prototype Transformer: Towards Language Model Architectures Interpretable by Design

University of Cambridge(剑桥大学);ETH Zurich(苏黎世联邦理工学院)

Yordan Yordanov, Matteo Forasassi, Bayar Menzat, Ruizhi Wang, Chang Qi, Markus Kaltenberger, Amine M'Charrak, Tommaso Salvatori, Thomas Lukasiewicz

AI总结:

提出原型Transformer(ProtoT),一种用线性代价原型模块替代二次代价自注意力的自回归语言模型架构,原型自动捕获可命名概念,提升可解释性并支持行为编辑。

摘要:

尽管最先进的语言模型(LM)在某些领域超越了大多数人类,但其推理过程仍然不透明,降低了信任度并增加了欺骗和幻觉的风险。我们引入了原型Transformer(ProtoT),一种自回归LM架构,它将Transformer的二次代价自注意力模块替换为基于原型的线性代价模块,原型是学习到的参数向量。在ProtoT中,原型创建了在不同时间尺度上聚合上下文信息的通信通道。我们表明,这种结构导致原型在训练过程中自动捕获可命名的概念,例如“女人”,为解释模型推理和对模型行为进行有针对性的编辑提供了途径。与基线相比,ProtoT在模型和数据规模上具有良好的扩展性,对输入扰动具有鲁棒性,并在文本生成和下游任务(包括GLUE)上表现良好。这些结果表明,ProtoT是朝着设计上更可解释的自回归语言模型迈出的有希望的一步。

英文摘要:

While state-of-the-art language models (LMs) surpass most humans in certain domains, their reasoning remains largely opaque, reducing trust and increasing the risk of deception and hallucination. We introduce the Prototype Transformer (ProtoT), an autoregressive LM architecture that replaces the quadratic-cost self-attention module of the Transformer with a linear-cost module based on prototypes, which are learned parameter vectors. In ProtoT, prototypes create communication channels that aggregate contextual information at different time scales. We show that this structure leads prototypes to automatically capture nameable concepts, such as "woman", during training, offering a path toward interpreting model reasoning and making targeted edits to model behavior. Compared with baselines, ProtoT scales well with model and data size, is robust to input perturbations, and performs well on text generation and downstream tasks, including GLUE. These results suggest that ProtoT is a promising step toward autoregressive language models that are more interpretable by design.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑