arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.18205cs.CL
收藏

上下文结构化标记依赖编码用于大型语言模型

Contextually Structured Token Dependency Encoding for Large Language Models

James Blades, Frederick Somerfield, William Langley, Susan Everingham, Maurice Witherington

AI总结:

本文提出一种依赖感知的标记编码方法,通过依赖加权注意力显式嵌入句法和语义关系,以提升大型语言模型的上下文连贯性和层次一致性,实验显示困惑度降低且无需外部标注。

摘要:

大型神经架构中的标记表示策略通常依赖于上下文精炼的嵌入,然而传统方法很少在标记交互中显式编码结构化关系。自注意力机制能有效捕获动态上下文依赖,但其对学习权重分布的依赖限制了生成序列中长距离层次结构的保持。依赖感知的标记编码引入了一种结构化的嵌入初始化方法,确保关系约束嵌入到标记表示中,而非仅通过注意力动态推断。所提出的编码机制通过依赖加权的注意力计算细化标记交互,确保句法和语义依赖在多个处理层中得以保留。实证评估表明,在多种语言基准上困惑度有所降低,提示自回归文本生成中上下文连贯性和预测一致性得到改善。计算效率评估显示,由于编码模块中的额外矩阵计算,内存消耗和训练时间适度增加,但在传统Transformer架构内可扩展性仍然可行。结构化编码增强了词汇多样性和依赖保持,无需外部句法标注或辅助训练目标即可强化语言连贯性。统计比较显示依赖对齐有所改善,尤其是在较长序列中,传统自注意力模型在层次一致性上表现出退化。句子长度分布表明短语突变减少,进一步支持显式依赖编码促进更结构化短语生成的假设。

英文摘要:

Token representation strategies within large-scale neural architectures often rely on contextually refined embeddings, yet conventional approaches seldom encode structured relationships explicitly within token interactions. Self-attention mechanisms effectively capture dynamic contextual dependencies, but their reliance on learned weight distributions limits the preservation of long-range hierarchical structures in generated sequences. Dependency-aware token encoding introduces a structured approach to embedding initialization, ensuring that relational constraints are embedded within token representations rather than inferred solely through attention dynamics. The proposed encoding mechanism refines token interactions through dependency-weighted attention computations, ensuring that syntactic and semantic dependencies are retained across multiple processing layers. Empirical evaluations indicate reductions in perplexity across diverse linguistic benchmarks, suggesting improvements in contextual coherence and predictive consistency in autoregressive text generation. Computational efficiency assessments reveal a moderate increase in memory consumption and training time, attributed to additional matrix computations within the encoding module, yet scalability remains feasible within conventional transformer architectures. Structured encoding enhances lexical variation and dependency retention, reinforcing linguistic coherence without requiring external syntactic annotations or auxiliary training objectives. Statistical comparisons highlight improvements in dependency alignment, particularly in longer sequences where conventional self-attention models exhibit degradation in hierarchical consistency. Sentence length distributions indicate a reduction in abrupt phrase transitions, further supporting the hypothesis that explicit dependency encoding facilitates more structured phrase generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑