ADE:自适应词典嵌入——将多锚表示扩展到大语言模型
ADE: Adaptive Dictionary Embeddings -- Scaling Multi-Anchor Representations to Large Language Models
AI总结:
ADE通过引入词汇投影、分组位置编码和上下文感知锚重加权,有效扩展多锚表示到大语言模型,减少参数并提升性能。
摘要:
词嵌入是自然语言处理的基础,但传统方法用单个向量表示每个词,导致多义词表示瓶颈和语义表达受限。多锚表示通过组合多个向量表示词,但受限于计算效率和与现代Transformer架构的整合不足。我们引入自适应词典嵌入(ADE),一种将多锚词表示扩展到大语言模型的框架。ADE有三个关键贡献:(1)词汇投影(VP),将昂贵的两阶段锚查找转换为单个高效矩阵运算;(2)分组位置编码(GPE),一种新颖的位置编码方案,其中同一词的锚共享位置信息,保持语义一致性的同时允许锚级变化;(3)上下文感知锚重加权,利用自注意力动态组合锚贡献。我们将这些组件整合到段落感知Transformer(SAT)中,为推理时提供上下文感知的锚贡献重加权。我们在AG News和DBpedia-14文本分类基准上评估ADE。与DeBERTa-v3-base相比,ADE训练参数减少98.7%,在DBpedia-14上超越DeBERTa(98.06% vs. 97.80%),在AG News上接近DeBERTa(90.64% vs. 94.50%),同时将嵌入层压缩了40倍——证明多锚表示是现代Transformer架构中单向量嵌入的实用且参数高效的替代方案。
英文摘要:
Word embeddings are fundamental to natural language processing, yet traditional approaches represent each word with a single vector, creating representational bottlenecks for polysemous words and limiting semantic expressiveness. While multi-anchor representations have shown promise by representing words as combinations of multiple vectors, they have been limited to small-scale models due to computational inefficiency and lack of integration with modern transformer architectures. We introduce Adaptive Dictionary Embeddings (ADE), a framework that successfully scales multi-anchor word representations to large language models. ADE makes three key contributions: (1) Vocabulary Projection (VP), which transforms the costly two-stage anchor lookup into a single efficient matrix operation; (2) Grouped Positional Encoding (GPE), a novel positional encoding scheme where anchors of the same word share positional information, preserving semantic coherence while enabling anchor-level variation; and (3) context-aware anchor reweighting, which leverages self-attention to dynamically compose anchor contributions based on sequence context. We integrate these components into the Segment-Aware Transformer (SAT), which provides context-aware reweighting of anchor contributions at inference time. We evaluate ADE on AG News and DBpedia-14 text classification benchmarks. With 98.7% fewer trainable parameters than DeBERTa-v3-base, ADE surpasses DeBERTa on DBpedia-14 (98.06% vs. 97.80%) and approaches it on AG News (90.64% vs. 94.50%), while compressing the embedding layer over 40x -- demonstrating that multi-anchor representations are a practical and parameter-efficient alternative to single-vector embeddings in modern transformer architectures.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。