发表机构
Boston University（波士顿大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究对比LLMs从规则与示例中进行上下文学习的效能，发现规则学习更可靠，指令调优可放大其优势，代数任务中规则增益更显著，分布敏感或依赖参数知识的任务中优势较小。
AI 中文摘要
大型语言模型（LLMs）具备上下文学习能力，无需更新模型权重即可从提示上下文学习新任务。我们对比了两种主流上下文学习模式的学习效能：（1）从规则描述中学习（指令跟随）；（2）从输入输出演示示例中学习（少样本提示）。通过覆盖游戏、算术、语言推理等不同领域的5项学习任务，我们对比了两种学习模式（规则vs.示例）在指定同一基础任务时的表现，还探究了影响学习效能的模型与任务属性。研究发现，模型通常从规则中学习的可靠性高于仅从示例中学习；在规则之外增加示例，或单纯扩大示例数量，无法带来一致且显著的提升。指令调优会放大基于规则学习的优势，同时保持基于示例学习的能力不受影响。令人惊讶的是，我们发现基础模型中不存在基于示例学习的特权效应，且在代数任务领域，规则仍能带来增益。总体而言，当任务涉及代数抽象与计算时，规则相较于示例的效能优势更大；当任务需要分布敏感性和/或依赖参数知识时，该优势更小。
英文摘要
Large language models (LLMs) exhibit in-context learning capabilities, where they can learn new tasks from prompt contexts without weight updates. We compare the learning efficacies of two prominent modes of in-context learning: (1) learning from descriptions of rules (instruction following); and (2) learning from examples of input-output demonstrations (few-shot prompting). Through five learning tasks that cover diverse domains (games, arithmetic, linguistic inferences), we compare two modes of learning (rules vs. examples) specifying the same underlying task. We furthermore explore model and task properties that modulate the learning efficacies. We find that models generally learn more reliably from rules than from examples alone, and additional examples on top of rules or simply scaling up the number of examples do not lead to consistent and significant gains. Instruction tuning amplifies the benefit of rule-based learning while keeping example-based learning capacities intact. Surprisingly, we find no privileged effect of example-based learning in base models, and rules still lead to gains in algebraic task domains. Overall, the comparative efficacy of rules over examples is larger when the task recruits algebraic abstractions and computations, and smaller when the task requires distributional sensitivity and/or recruits parametric knowledge.
