Large language models (LLMs) have a substantial capacity for high-level analogical reasoning: reproducing patterns in linear text that occur in their training data (zero-shot evaluation) or in the provided context (few-shot in-context learning). However, recent studies show that even the more advanced LLMs fail in scenarios that require reasoning over multiple objects or facts and making sequences of logical deductions. We propose a two-stage probabilistic inference paradigm, ThinkSum, which reasons over sets of objects or facts in a structured manner. In the first stage (Think - retrieval of associations), a LLM is queried in parallel over a set of phrases extracted from the prompt or an auxiliary model call. In the second stage (Sum - probabilistic inference or reasoning), the results of these queries are aggregated to make the final prediction. We demonstrate the possibilities and advantages of ThinkSum on the BIG-bench suite of LLM evaluation tasks, achieving improvements over the state of the art using GPT-family models on thirteen difficult tasks, often with far smaller model variants. We also compare and contrast ThinkSum with other proposed modifications to direct prompting of LLMs, such as variants of chain-of-thought prompting. Our results suggest that because the probabilistic inference in ThinkSum is performed outside of calls to the LLM, ThinkSum is less sensitive to prompt design, yields more interpretable predictions, and can be flexibly combined with latent variable models to extract structured knowledge from LLMs. Overall, our proposed paradigm represents a promising approach for enhancing the reasoning capabilities of LLMs.
Humans and LLMs Diverge on Probabilistic Inferences
McGill University（麦吉尔大学）；Mila – Quebec AI Institute（魁北克人工智能研究所）；University of Vienna（维也纳大学）；FNRS – UCLouvain（佛罗伦萨-鲁汶大学）；Canada CIFAR AI Chair（加拿大CIFAR人工智能主席）
Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models
MIT（麻省理工学院）；Meta；Google（谷歌公司）；DeepMind University of British Columbia（不列颠哥伦比亚大学深度思维学院）；Vector Institute（向量研究所）；New York University（纽约大学）；Google Research（谷歌研究）
StruProKGR: A Structural and Probabilistic Framework for Sparse Knowledge Graph Reasoning
CAS Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院网络数据科学与技术重点实验室，计算技术研究所）；School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）
