LACE:用于跨线探索的晶格注意力
LACE: Lattice Attention for Cross-thread Exploration
AI总结:
LACE通过引入跨线注意力机制,使大语言模型在推理过程中实现协同合作,提升推理准确性达7个百分点。
摘要:
当前大型语言模型在推理过程中是孤立运作的。尽管常见于并行采样多个推理路径,但这些轨迹不相互作用,往往以相同的方式失败。我们引入LACE,一种将推理从独立尝试转变为协调并行过程的框架。通过重新利用模型架构以启用跨线注意力,LACE允许并发推理路径共享中间见解并相互纠正。核心挑战是缺乏自然训练数据,这种协作行为显而易见。我们通过合成数据管道显式教授模型在不同线程间沟通和纠错。实验表明,这种统一探索显著优于标准并行搜索,提高了推理准确性超过7个百分点。我们的结果表明,当允许并行推理路径相互作用时,大型语言模型可以更有效。
英文摘要:
Current large language models reason in isolation. Although it is common to sample multiple reasoning paths in parallel, these trajectories do not interact, and often fail in the same redundant ways. We introduce LACE, a framework that transforms reasoning from a collection of independent trials into a coordinated, parallel process. By repurposing the model architecture to enable cross-thread attention, LACE allows concurrent reasoning paths to share intermediate insights and correct one another during inference. A central challenge is the absence of natural training data that exhibits such collaborative behavior. We address this gap with a synthetic data pipeline that explicitly teaches models to communicate and error-correct across threads. Experiments show that this unified exploration substantially outperforms standard parallel search, improving reasoning accuracy by over 7 points. Our results suggest that large language models can be more effective when parallel reasoning paths are allowed to interact.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。