arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.23985cs.AIcs.CLcs.LG
收藏

表征曲率调节大型语言模型中的行为不确定性

Representational Curvature Modulates Behavioral Uncertainty in Large Language Models

Brain and Cognitive Sciences(脑与认知科学);Massachusetts Institute of Technology(麻省理工学院)

Jack King, Evelina Fedorenko, Eghbal A. Hosseini

AI总结:

研究通过分析表征曲率与token级熵的关系,揭示了大型语言模型中行为不确定性的调节机制,发现曲率变化可影响预测熵。

摘要:

在自回归大型语言模型(LLMs)中,时间直化提供了下一token预测目标如何塑造表示的解释。模型在各层学习逐步直化输入序列的表示轨迹,可能通过线性外推促进下一token预测。然而,轨迹与token级行为之间的直接联系一直缺失。我们通过将上下文曲率——表示轨迹在最近上下文内弯曲程度的几何度量——与下一token熵相关联,提供了这种联系。在两个模型(GPT-2 XL和Pythia-2.8B)中,上下文曲率与熵相关联,并在训练过程中出现。扰动实验揭示了选择性依赖:通过轨迹对齐的干预操纵曲率可可靠地调节熵,而几何上不匹配的扰动无影响。最后,训练过程中正则化表示使其更直,可略微减少token级熵而不影响验证损失。这些结果识别了轨迹曲率作为任务对齐的表示特征,影响LLM中的行为不确定性。

英文摘要:

In autoregressive large language models (LLMs), temporal straightening offers an account of how the next-token prediction objective shapes representations. Models learn to progressively straighten the representational trajectory of input sequences across layers, potentially facilitating next-token prediction via linear extrapolation. However, a direct link between this trajectory and token-level behavior has been missing. We provide such a link by relating contextual curvature-a geometric measure of how sharply the representational trajectory bends over recent context-to next-token entropy. Across two models (GPT-2 XL and Pythia-2.8B), contextual curvature is correlated with entropy, and this relationship emerges during training. Perturbation experiments reveal selective dependence: manipulating curvature through trajectory-aligned interventions reliably modulates entropy, while geometrically misaligned perturbations have no effect. Finally, regularizing representations to be straighter during training modestly reduces token-level entropy without degrading validation loss. These results identify trajectory curvature as a task-aligned representational feature that influences behavioral uncertainty in LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑