发表机构
Tsinghua University（清华大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究提出Trace as State方法，将推理轨迹置于长上下文Transformer的上下文前，在27组实验组合中26组优于对照方法，显著提升了模型在长上下文推理任务上的表现。
AI 中文摘要
Transformer以因果方式处理信息，但长上下文推理可能依赖于仅在后续才能发现的任务状态。我们通过条件状态更新任务来形式化这种不匹配。对于因果状态更新处理器，在最坏情况下，先提供条件所需的内存比后提供条件少指数级。受此原理启发，我们提出了Trace as State（轨迹即状态）方法：将收集到的推理轨迹作为任务状态的文本代理，在重新处理时将其置于长上下文块之前，使先前推导的信息能指导重读。我们对Trace as State和匹配对照方法Trace Append（使用相同任务状态代理但将其置于上下文之后）进行了大量实验，涉及3种模型和3个长上下文数据集。在报告的27种模型、任务和指标组合中，Trace as State在26种组合上优于Trace Append。在GraphWalks Parents任务上，精确匹配指标显示，DeepSeek V4 Pro Preview的首次通过准确率为29.2%，使用Trace Append时为43.0%，使用Trace as State时提升至81.8%；GLM-5.2的对应准确率从66.4%和83.2%提升至100.0%。这些结果表明，将轨迹置于上下文之前可在保留因果Transformer结构的同时提升长上下文推理能力。
英文摘要
Transformers process information causally, but long-context reasoning may depend on task state discovered only later. We formalize this mismatch through conditional state update tasks. For causal state update processors, providing the condition first can require exponentially less memory in the worst case than providing it last. Motivated by this principle, we introduce Trace as State. We use collected reasoning traces as a textual proxy for task state and place it before the long-context block on a fresh pass, allowing information derived previously to guide rereading. We conduct extensive experiments on Trace as State and Trace Append, a matched control that uses the same task state proxy but put it after the context. Across three models and three long-context datasets, Trace as State outperforms Trace Append in 26 of 27 reported combinations of model, task, and metric. On GraphWalks Parents, exact match lifts DeepSeek V4 Pro Preview from 29.2% on the initial pass and 43.0% with Trace Appendto 81.8% with Trace as State, and from 66.4% and 83.2% to 100.0% for GLM-5.2. These results show that placing traces before the context can improve long-context reasoning while retaining the causal transformer structure.
Commentspreprint
