arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.17617cs.CL
收藏

使用概率层重对齐的大型语言模型结构化上下文重组

Structured Context Recomposition for Large Language Models Using Probabilistic Layer Realignment

Jonathan Teel, Jocasta Cumberbatch, Raphael Benington, Quentin Baskerville

AI总结:

本文提出SCR方法,通过概率层重对齐策略动态调整transformer层内表示,利用递归加权函数增强长序列生成的上下文一致性与推理稳定性,且计算开销可控。

摘要:

扩展序列生成通常会导致上下文一致性下降,因为传统的自注意力机制无法有效保留长程依赖。现有方法(包括内存压缩和检索增强条件化)引入了计算权衡,要么增加推理延迟,要么施加额外存储开销。Structured Context Recomposition (SCR) 引入了一种概率层重对齐策略,在 transformer 层内动态调整学习到的表示,确保语义相关的嵌入在扩展转换过程中持续存在。该方法通过递归加权函数增强一致性保留,该函数根据推断的上下文相关性重新分配表示重点,而非依赖固定的 token 级注意力分数。实验结果表明,概率重对齐缓解了突变的话题转移和逻辑不一致,特别是在序列超过标准注意力窗口约束的场景中。序列级熵分析进一步显示,SCR 调节了表示变异性,且未引入过度的输出正则化,使模型在保持上下文对齐的同时维持生成多样性。注意力头偏差测量证实,分层重加权有助于在 transformer 层间实现更平滑的 token 依赖转换,增强了多轮交互和文档级推理的稳定性。计算资源评估表明,尽管 SCR 导致处理时间适度增加,但内存开销仍保持在可行范围内,使其适合在自回归生成应用中实际部署。

英文摘要:

Extended sequence generation often leads to degradation in contextual consistency due to the inability of conventional self-attention mechanisms to effectively retain long-range dependencies. Existing approaches, including memory compression and retrieval-augmented conditioning, introduce computational trade-offs that either increase inference latency or impose additional storage overhead. Structured Context Recomposition (SCR) introduces a probabilistic layer realignment strategy that dynamically adjusts learned representations within transformer layers, ensuring that semantically relevant embeddings persist throughout extended transformations. The proposed method enhances coherence retention through a recursive weighting function that redistributes representational emphasis based on inferred contextual relevance rather than relying on fixed token-level attention scores. Empirical results indicate that probabilistic realignment mitigates abrupt topic shifts and logical inconsistencies, particularly in scenarios where sequences exceed standard attention window constraints. Sequence-level entropy analysis further reveals that SCR moderates representational variability without introducing excessive output regularization, allowing models to sustain generative diversity while preserving contextual alignment. Attention head deviation measurements confirm that hierarchical reweighting contributes to smoother token dependency transitions across transformer layers, reinforcing the stability of multi-turn interactions and document-level reasoning. Computational resource assessments show that while SCR incurs a moderate increase in processing time, memory overhead remains within feasible limits, making it suitable for practical deployment in autoregressive generative applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑