arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.09253cs.LG
收藏

通过长度感知注意力先验和增益感知训练实现固定测试时间成本下的高效推理

Efficient Reasoning at Fixed Test-Time Cost via Length-Aware Attention Priors and Gain-Aware Training

Rian Atri

AI总结:

本文提出长度感知注意力先验和增益感知训练方法,通过减少验证交叉熵并保持测试时间成本不变,在长跨度噪声logit区域实现高效推理。

摘要:

我们研究在计算受限条件下如何高效推理。我们探讨如何在不增加测试时间成本的情况下做出结构化且正确的决策。我们为小型和中型Transformer添加了两个仅在训练时使用的组件,这些组件也适用于更广泛的可微优化器。首先,通过模糊模式位置对齐构建的长度感知注意力先验(RPA)产生一个归一化的softmax前偏置,该偏置在不引入新推理参数的情况下引导注意力,如同一个结构化正则化器。其次,一个最小增益感知控制器Guardian仅在验证改进值得时调整注意力的锐度,遵循非凸优化的双时间尺度策略梯度观点。在推理时,该控制器被禁用。从KL视角来看,softmax of z plus log pi作为MAP在KL正则化下得到支撑,将先验建立在原理性目标上。在WikiText 2上严格计算平衡条件下,我们减少了验证交叉熵,同时保持基线的延迟和内存。在推理时,我们添加一个预计算、缓存的先验B of T作为每个头部的单个加法偏置。控制器不运行。实际上,这带来的开销极小,仅是每个头部的缓存偏置加法,没有可测量的p50延迟变化。我们的结果表明,长度感知先验和后期阶段增益控制在保持测试时间成本不变的同时,保留了稀缺的改进,特别是在长跨度、噪声logit区域。

英文摘要:

We study efficient reasoning under tight compute. We ask how to make structured, correct decisions without increasing test time cost. We add two training only components to small and medium Transformers that also transfer to broader differentiable optimizers. First, a length aware attention prior built via fuzzy regime position alignment, RPA, yields a normalized pre softmax bias that guides attention like a structured regularizer while adding no new inference parameters. Second, a minimal gain aware controller, Guardian, nudges attention sharpness only when validation improvements warrant it, following a two timescale policy gradient view of nonconvex optimization. It is disabled at inference. A KL perspective shows softmax of z plus log pi as MAP with KL regularization, grounding the prior in a principled objective. Under strict compute parity on WikiText 2, we reduce validation cross entropy while matching baseline latency and memory. At inference, we add a precomputed, cached prior B of T as a single additive bias per head. The controller does not run. In practice, this incurs negligible overhead, a cached bias add per head, with no measurable p50 latency shift. Our results suggest that length aware priors and late phase gain control preserve scarce improvements, especially in long span, noisy logit regimes, while keeping test time costs effectively unchanged.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑