评估大型语言模型对指令的鲁棒性
Evaluating the Robustness to Instructions of Large Language Models
AI总结：
本文评估了Alpaca、Vicuna等指令微调大型语言模型在已见和未见任务上的鲁棒性，发现模型处理不熟悉指令的性能显著下降，且对关系抽取指令的鲁棒性弱于问答指令。
摘要：
最近，指令微调作为一种增强大型语言模型（LLMs）在新任务上零样本能力的潜在方法而备受瞩目。该技术展现出提升中等规模LLMs性能的卓越能力，有时甚至能达到与更大模型变体相媲美的性能水平。本文重点关注经过指令微调的LLMs在已见和未见任务上的鲁棒性。我们以真实世界的关系抽取数据集为案例，对包括Alpaca、Vicuna、WizardLM和传统任务导向模型（Flan-T5-XL/XXL, T0++）在内的六个模型进行了探索。我们对这些基于开放域指令和任务导向指令微调的指令遵循LLMs进行了全面评估。主要讨论了它们的性能及对指令的鲁棒性。我们观察到，在大多数情况下，模型处理不熟悉指令的性能往往会显著恶化，并且模型对RE指令的鲁棒性相比QA有所下降。此外，我们发现直到达到特定的参数规模阈值（3B）之前，FLAN-T5模型的性能随着参数量的增加而提升。不同规模的FLAN-T5模型对RE指令的鲁棒性均差于对QA指令的鲁棒性。
英文摘要：
Recently, Instruction fine-tuning has risen to prominence as a potential method for enhancing the zero-shot capabilities of Large Language Models (LLMs) on novel tasks. This technique has shown an exceptional ability to boost the performance of moderately sized LLMs, sometimes even reaching performance levels comparable to those of much larger model variants. The focus is on the robustness of instruction-tuned LLMs to seen and unseen tasks. We conducted an exploration of six models including Alpaca, Vicuna, WizardLM, and Traditional Task-oriented Models(Flan-T5-XL/XXL, T0++) using real-world relation extraction datasets as case studies. We carried out a comprehensive evaluation of these instruction-following LLMs which have been tuned based on open-domain instructions and task-oriented instructions. The main discussion is their performance and robustness towards instructions. We have observed that in most cases, the model's performance in dealing with unfamiliar instructions tends to worsen significantly, and the robustness of the model for RE instructions deteriorates compared to QA. Further, we discovered that up until a certain parameter size threshold (3B), the performance of the FLAN-T5 model improves as the parameter count increases. The robustness of different scales of FLAN-T5 models to RE instruction is worse than the robustness to QA instruction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
