arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.09614cs.CLcs.AI
收藏

生成式大语言模型对不确定文本的推理

Reasoning over Uncertain Text by Generative Large Language Models

Michigan State University(密歇根州立大学);Florida Institute for Human and Machine Cognition(佛罗里达人类与机器认知研究所);University of West Florida(西佛罗里达大学)

Aliakbar Nafar, Kristen Brent Venable, Parisa Kordjamshidi

AI总结:

本文针对大语言模型概率推理能力不足的问题,提出贝叶斯语言推理数据集BLInD及多种形式化提示策略,并通过实验验证了这些策略在多个LLMs上的有效性。

摘要:

本文探讨了大语言模型(LLMs)在推理包含以概率值明确量化的不确定性信息的文本时所面临的挑战。这类推理与从日常对话到医疗决策等多种情境相关。尽管LLMs在数学推理能力上有所提升,但它们在概率推理方面仍表现出显著困难。为解决这一问题,我们引入了贝叶斯语言推理数据集(BLInD),这是一个专门设计用于测试LLMs概率推理能力的新数据集。我们利用BLInD找出LLMs在涉及概率推理任务中的局限性。此外,我们提出了几种提示策略,将问题映射到不同的形式化表示,包括Python代码、概率算法和概率逻辑编程。最后,我们对所提方法在BLInD以及一个因果推理问答数据集的改编版本上进行了评估。我们的实证结果凸显了所提策略在多个LLMs上的有效性。

英文摘要:

This paper considers the challenges Large Language Models (LLMs) face when reasoning over text that includes information involving uncertainty explicitly quantified via probability values. This type of reasoning is relevant to a variety of contexts ranging from everyday conversations to medical decision-making. Despite improvements in the mathematical reasoning capabilities of LLMs, they still exhibit significant difficulties when it comes to probabilistic reasoning. To deal with this problem, we introduce the Bayesian Linguistic Inference Dataset (BLInD), a new dataset specifically designed to test the probabilistic reasoning capabilities of LLMs. We use BLInD to find out the limitations of LLMs for tasks involving probabilistic reasoning. In addition, we present several prompting strategies that map the problem to different formal representations, including Python code, probabilistic algorithms, and probabilistic logical programming. We conclude by providing an evaluation of our methods on BLInD and an adaptation of a causal reasoning question-answering dataset. Our empirical results highlight the effectiveness of our proposed strategies for multiple LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑