稀疏门控语言模型中的路由响应上下文
Routing in Sparsely-gated Language Models responds to Context
AI总结：
本文通过追踪相似度标注文本对的路由决策评估了稀疏门控语言模型中 token-专家分配的上下文敏感性，发现编码器层路由主要依赖语义关联但受上下文细化，而解码器层路由变异性更大且对上下文不敏感。
摘要：
语言模型（LMs）近期引入了由路由器和一组专家组成的混合专家层，以在固定计算预算下扩大参数量。基于先前表明 token-专家分配主要受 token 标识和位置影响的研究，我们追踪了相似度标注文本对的路由决策，以评估学习到的 token-专家分配的上下文敏感性。我们观察到，编码器层的路由主要依赖于（语义）关联，但上下文线索提供了额外的细化层。相反，解码器层的路由更具多变性，且对上下文的敏感性显著较低。
英文摘要：
Language Models (LMs) recently incorporate mixture-of-experts layers consisting of a router and a collection of experts to scale up their parameter count given a fixed computational budget. Building on previous efforts indicating that token-expert assignments are predominantly influenced by token identities and positions, we trace routing decisions of similarity-annotated text pairs to evaluate the context sensitivity of learned token-expert assignments. We observe that routing in encoder layers mainly depends on (semantic) associations, but contextual cues provide an additional layer of refinement. Conversely, routing in decoder layers is more variable and markedly less sensitive to context.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
