arXiv 2403.042332024-03cs.CLcs.AI

TEGEE：任务定义引导的专家集成用于泛化与少样本学习

TEGEE: Task dEfinition Guided Expert Ensembling for Generalizable and Few-shot Learning

HKUST（香港科技大学）；University of Manchester（曼彻斯特大学）；Institute of Automation, CAS（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；M-A-P；Beijing University of Posts and Telecommunications（北京邮电大学）；Ohio State University（俄亥俄州立大学）；University of Waterloo（滑铁卢大学）；Peking University（北京大学）；BAAI（北京智源人工智能研究院）

Xingwei Qu, Yiming Liang, Yucheng Wang, Tianyu Zheng, Tommy Yue, Xingyuan Bu, Lei Ma, Stephen W. Huang, Jiajun Zhang, Yinan Shi, Chenghua Lin, Jie Fu, Ge Zhang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对LLM上下文学习中的任务提取与定义哪个更关键的问题，提出TEGEE方法，采用双3B模型分别提取任务定义和从演示中学习，性能媲美LLaMA2-13B，并支持多样本与持续学习。

摘要：

大型语言模型（LLMs）展现出进行上下文学习（ICL）的能力，即直接从演示中提供的示例中获取新任务。这一过程被认为通过一种隐式的任务选择机制运作，该机制涉及从这些演示中提取和处理任务定义。然而，关键问题仍然存在：任务提取和定义哪个更为重要？以及如何进一步改进这些能力？为了解决这些问题，我们提出了TEGEE（任务定义引导的专家集成），一种显式提取任务定义并根据特定任务生成响应的方法。我们的框架采用双3B模型方法，每个模型分配不同的角色：一个专注于任务定义提取，另一个处理从演示中学习。这种模块化方法支持了提取任务定义比处理任务本身更为重要的假设。实证评估表明，TEGEE的性能与更大的LLaMA2-13B模型相当。通过利用模块化设计，我们的方法将传统的ICL从少样本学习扩展到多样本学习，支持无限数量的演示，并增强了持续学习能力。

英文摘要：

Large Language Models (LLMs) exhibit the ability to perform in-context learning (ICL), where they acquire new tasks directly from examples provided in demonstrations. This process is thought to operate through an implicit task selection mechanism that involves extracting and processing task definitions from these demonstrations. However, critical questions remain: Which is more essential -- task extraction or definition? And how can these capabilities be further improved? To address these questions, we propose \textbf{TEGEE} (Task Definition Guided Expert Ensembling), a method that explicitly extracts task definitions and generates responses based on specific tasks. Our framework employs a dual 3B model approach, with each model assigned a distinct role: one focuses on task definition extraction, while the other handles learning from demonstrations. This modular approach supports the hypothesis that extracting task definitions is more vital than processing the task itself. Empirical evaluations show that TEGEE performs comparably to the larger LLaMA2-13B model. By leveraging a modular design, our approach extends traditional ICL from few-shot to many-shot learning, supporting an unlimited number of demonstrations and enhancing continual learning capabilities.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。