arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.09849cs.CLcs.AI
收藏

自一致性提升数学推理的校准能力

Self-Consistency Boosts Calibration for Math Reasoning

School of Informatics, Xiamen University(厦门大学信息学院);Tencent AI Lab(腾讯AI实验室)

Ante Wang, Linfeng Song, Ye Tian, Baolin Peng, Lifeng Jin, Haitao Mi, Jinsong Su, Dong Yu

AI总结:

针对数学推理任务,提出基于自一致性的三种即插即用校准方法,在GSM8K和MathQA上优于现有p(True)或logit方法,更好桥接置信度与准确性。

摘要:

校准建立了准确性与模型置信度之间的关联,对大型语言模型(LLM)的开发至关重要。我们基于自一致性(Wang等人,2022)设计了三种即插即用的校准方法,用于数学推理任务。在使用强大的开源LLM(Mistral和LLaMA2)对两个流行基准(GSM8K和MathQA)进行的评估中,我们的方法在桥接模型置信度与准确性方面优于现有基于p(True)(Kadavath等人,2022)或logit(Kadavath等人,2022)的方法。

英文摘要:

Calibration, which establishes the correlation between accuracy and model confidence, is important for LLM development. We design three off-the-shelf calibration methods based on self-consistency (Wang et al., 2022) for math reasoning tasks. Evaluation on two popular benchmarks (GSM8K and MathQA) using strong open-source LLMs (Mistral and LLaMA2), our methods better bridge model confidence and accuracy than existing methods based on p(True) (Kadavath et al., 2022) or logit (Kadavath et al., 2022).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑