Modern language models (LMs) can learn to perform new tasks in different ways: in instruction following, the target task is described explicitly in natural language; in few-shot prompting, the task is specified implicitly with a small number of examples; in instruction inference, LMs are presented with in-context examples and are then prompted to generate a natural language task description before making predictions. Each of these procedures may be thought of as invoking a different form of reasoning: instruction following involves deductive reasoning, few-shot prompting involves inductive reasoning, and instruction inference involves abductive reasoning. How do these different capabilities relate? Across four LMs (from the gpt and llama families) and two learning problems (involving arithmetic functions and machine translation) we find a strong dissociation between the different types of reasoning: LMs can sometimes learn effectively from few-shot prompts even when they are unable to explain their own prediction rules; conversely, they sometimes infer useful task descriptions while completely failing to learn from human-generated descriptions of the same task. Our results highlight the non-systematic nature of reasoning even in some of today's largest LMs, and underscore the fact that very different learning mechanisms may be invoked by seemingly similar prompting procedures.
Inferring Latent Intentions: Attributional Natural Language Inference in LLM Agents
University of Manchester（曼彻斯特大学）；University of Sheffield（谢菲尔德大学）；Idiap Research Institute（Idiap研究 institute）；National Biomarker Centre, CRUK-MI, University of Manchester（国家生物标志物中心，CRUK-MI，曼彻斯特大学）
Assessing the Reasoning Capabilities of LLMs in the context of Evidence-based Claim Verification
University of Warwick（沃里克大学）；Queen Mary University of London（伦敦大学女王学院）；University of Bologna（博洛尼亚大学）；King’s College London（伦敦大学国王学院）；The Alan Turing Institute（艾伦·图灵研究所）
The Role of Deductive and Inductive Reasoning in Large Language Models
University of Edinburgh（爱丁堡大学）；University of Manchester（曼彻斯特大学）；University of Washington（华盛顿大学）；Tsinghua University（清华大学）；Skai Intelligence（Skai智能科技）；University of Copenhagen（哥本哈根大学）
MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Can Large Language Models emulate an inductive Thematic Analysis of semi-structured interviews? An exploration and provocation on the limits of the approach and the model
Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinement
University of Manchester（曼彻斯特大学）；Idiap Research Institute（Idiap研究所）；National Biomarker Centre, CRUK-MI, University of Manchester（曼彻斯特大学英国癌症研究中心曼彻斯特研究所国家生物标志物中心）
R-APS: Compositional Reasoning and In-Context Meta-Learning for Constrained Design via Reflective Adversarial Pareto Search
Idiap Research Institute（Idiap研究 institute）；École Polytechnique Fédérale de Lausanne（瑞士联邦理工学院）；Honda Research Institute Europe（本田欧洲研究机构）；Department of Computer Science, University of Manchester（曼彻斯特大学计算机科学系）；National Biomarker Centre, CRUK-MI, University of Manchester（曼彻斯特大学国家生物标志物中心）
Shuttle Between the Instructions and the Parameters of Large Language Models
The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（认知与决策智能复杂系统实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Xiaohongshu Inc（小红书公司）；Kuaishou Technology（快手技术）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
