arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.13486cs.CLcs.AI
收藏

注意指令:基于提示的学习中一致性与交互性的整体评估

Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learning

University Pompeu Fabra(庞培法布拉大学);Osnabrück University(奥斯纳布吕克大学);FAIR(Facebook人工智能研究院)

Lucas Weber, Elia Bruni, Dieuwke Hupkes

AI总结:

本文系统评估了提示学习中影响LLM预测稳定性的各项设计因素,通过在原版和指令微调模型上测试因素组合,揭示了不同因素的交互影响及使用建议。

摘要:

寻找将预训练语言模型适配到任务的最佳方法是当前NLP的一大挑战。正如上一代任务微调(TT)模型一样,通过上下文学习(ICL)适配任务的模型在某些设置中鲁棒,而在其他设置中则不然。在此,我们对哪些设计选择导致LLM预测的不稳定和不一致进行了详细分析。首先,我们展示了输入分布与标签间的虚假相关性——这是TT模型的已知问题——对提示模型仅构成次要问题。随后,我们对影响提示设置中预测的不同因素进行了系统、整体评估。我们在不同规模的原版和指令微调(IT)LLM上测试了一系列因素的所有可能组合,并对结果进行统计分析,以揭示哪些因素最具影响力、交互性或稳定性。结果表明,哪些因素在大多数设置中可无需防范直接使用,哪些应避免或谨慎处理。

英文摘要:

Finding the best way of adapting pre-trained language models to a task is a big challenge in current NLP. Just like the previous generation of task-tuned models (TT), models that are adapted to tasks via in-context-learning (ICL) are robust in some setups but not in others. Here, we present a detailed analysis of which design choices cause instabilities and inconsistencies in LLM predictions. First, we show how spurious correlations between input distributions and labels -- a known issue in TT models -- form only a minor problem for prompted models. Then, we engage in a systematic, holistic evaluation of different factors that have been found to influence predictions in a prompting setup. We test all possible combinations of a range of factors on both vanilla and instruction-tuned (IT) LLMs of different scale and statistically analyse the results to show which factors are the most influential, interactive or stable. Our results show which factors can be used without precautions and which should be avoided or handled with care in most settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑