arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.00205cs.CL
收藏

语言模型中的概念性与无偏推理

Conceptual and Unbiased Reasoning in Language Models

University of Pennsylvania(宾夕法尼亚大学);Tencent AI Lab(腾讯AI实验室)

Ben Zhou, Hongming Zhang, Sihao Chen, Dian Yu, Hongwei Wang, Baolin Peng, Dan Roth, Dong Yu

AI总结:

本文提出概念化框架以分析并提升大语言模型的抽象概念推理能力,发现现有模型相比直接推理下降9%-28%,并通过生成相似推理路径问题与自我精炼技术使性能提升8%-11%,从而构建更稳健且更少依赖归纳偏置的推理系统。

摘要:

概念推理,即从抽象和高层次视角进行推理的能力,是人类认知中泛化能力的关键。然而,关于大语言模型执行概念推理能力的研究仍然有限。在本工作中,我们弥合了这一空白,并提出了一种新颖的概念化框架,该框架迫使模型对抽象问题进行概念推理,并在可验证的符号空间中生成解决方案。利用该框架作为分析工具,我们表明现有的大语言模型在概念推理方面存在不足,与直接推理方法相比,在多个基准上的性能下降了9%到28%。随后,我们讨论了模型如何改进,因为高层次抽象推理是无偏且可泛化决策的关键。我们提出了两种技术,通过生成具有相似底层推理路径的熟悉问题并要求模型进行自我精炼,来添加可信的归纳信号。实验表明,我们提出的技术将模型的概念推理性能提高了8%到11%,实现了一个更稳健、更少依赖归纳偏置的推理系统。

英文摘要:

Conceptual reasoning, the ability to reason in abstract and high-level perspectives, is key to generalization in human cognition. However, limited study has been done on large language models' capability to perform conceptual reasoning. In this work, we bridge this gap and propose a novel conceptualization framework that forces models to perform conceptual reasoning on abstract questions and generate solutions in a verifiable symbolic space. Using this framework as an analytical tool, we show that existing large language models fall short on conceptual reasoning, dropping 9% to 28% on various benchmarks compared to direct inference methods. We then discuss how models can improve since high-level abstract reasoning is key to unbiased and generalizable decision-making. We propose two techniques to add trustworthy induction signals by generating familiar questions with similar underlying reasoning paths and asking models to perform self-refinement. Experiments show that our proposed techniques improve models' conceptual reasoning performance by 8% to 11%, achieving a more robust reasoning system that relies less on inductive biases.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑