DeLTa：基于Logit轨迹预测的解码策略提升事实性与推理能力
DeLTa: A Decoding Strategy based on Logit Trajectory Prediction Improves Factuality and Reasoning Ability
AI总结：
本文提出DeLTa解码策略，通过分析Transformer各层logit轨迹并应用线性回归调整下一token概率，在不修改模型架构与参数的前提下提升LLM事实准确性与推理能力，在多个基准上取得显著提升。
摘要：
大型语言模型（LLMs）在现实应用中的使用日益广泛。然而，对其生成内容可靠性的担忧持续存在，因为其经常偏离事实正确性或在逻辑推理方面表现出缺陷。本文提出一种新型解码策略，旨在增强事实准确性和推理能力，且无需对LLMs的架构或预训练参数进行任何修改。该方法通过分析Transformers中从低层到高层的logit轨迹并应用线性回归来调整下一个token的概率。我们发现，这种基于Logit轨迹的解码方法（DeLTa）能有效增强事实性和推理能力，同时减少错误生成。在TruthfulQA上的实验表明，DeLTa比基线提高了4.9%。此外，它在StrategyQA上提升了8.1%，在GSM8K上提升了7.3%，这两者都需要强大的推理能力。
英文摘要：
Large Language Models (LLMs) are increasingly being used in real-world applications. However, concerns about the reliability of the content they generate persist, as it frequently deviates from factual correctness or exhibits deficiencies in logical reasoning. This paper proposes a novel decoding strategy aimed at enhancing both factual accuracy and inferential reasoning without requiring any modifications to the architecture or pre-trained parameters of LLMs. Our approach adjusts next-token probabilities by analyzing the trajectory of logits from lower to higher layers in Transformers and applying linear regression. We find that this Decoding by Logit Trajectory-based approach (DeLTa) effectively reinforces factuality and reasoning while mitigating incorrect generation. Experiments on TruthfulQA demonstrate that DeLTa attains up to a 4.9% improvement over the baseline. Furthermore, it enhances performance by up to 8.1% on StrategyQA and 7.3% on GSM8K, both of which demand strong reasoning capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
