LLMs中的灾难性遗忘：跨语言任务的比较分析
Catastrophic Forgetting in LLMs: A Comparative Analysis Across Language Tasks
AI总结：
本研究评估了100亿参数以下的开源LLMs在GLUE基准测试上的持续微调，发现Phi-3.5-mini等模型在持续学习中表现出最小的遗忘和强大的学习能力，并强调了提示工程在优化模型性能中的作用。
摘要：
大型语言模型（LLMs）显著推进了自然语言处理（NLP），特别是在自然语言理解（NLU）任务中。随着我们迈向一个由基于LLM的智能体自主处理专业任务的智能体世界，这些模型在不遗忘先前学习信息的情况下适应新任务变得至关重要——这一挑战被称为灾难性遗忘。本研究评估了不同参数规模（特别是100亿参数以下的模型）的各种开源LLMs在GLUE基准测试的关键NLU任务上的持续微调，包括SST-2、MRPC、CoLA和MNLI。通过采用提示工程和特定任务调整，我们评估并比较了模型在学习新任务时保留先前知识的能力。我们的结果表明，像Phi-3.5-mini这样的模型在保持强大学习能力的同时表现出最小的遗忘，使其非常适合持续学习环境。此外，像Orca-2-7b和Qwen2.5-7B这样的模型在微调后展现出令人印象深刻的学习能力和整体性能。这项工作有助于理解LLMs中的灾难性遗忘，并强调了提示工程在优化持续学习场景下的模型性能方面的作用。
英文摘要：
Large Language Models (LLMs) have significantly advanced Natural Language Processing (NLP), particularly in Natural Language Understanding (NLU) tasks. As we progress toward an agentic world where LLM-based agents autonomously handle specialized tasks, it becomes crucial for these models to adapt to new tasks without forgetting previously learned information - a challenge known as catastrophic forgetting. This study evaluates the continual fine-tuning of various open-source LLMs with different parameter sizes (specifically models under 10 billion parameters) on key NLU tasks from the GLUE benchmark, including SST-2, MRPC, CoLA, and MNLI. By employing prompt engineering and task-specific adjustments, we assess and compare the models' abilities to retain prior knowledge while learning new tasks. Our results indicate that models such as Phi-3.5-mini exhibit minimal forgetting while maintaining strong learning capabilities, making them well-suited for continual learning environments. Additionally, models like Orca-2-7b and Qwen2.5-7B demonstrate impressive learning abilities and overall performance after fine-tuning. This work contributes to understanding catastrophic forgetting in LLMs and highlights prompting engineering to optimize model performance for continual learning scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
