arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.00256cs.AIcs.CLcs.LGstat.ML
收藏

一个提示并不足够:专家混合提示的自动化构建

One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Prompts

University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Korea Advanced Institute of Science & Technology(韩国科学技术院);Penn State University(宾夕法尼亚州立大学);University of Maryland, College Park(马里兰大学学院公园分校)

Ruochen Wang, Sohyun An, Minhao Cheng, Tianyi Zhou, Sung Ju Hwang, Cho-Jui Hsieh

AI总结:

本文提出Mixture-of-Prompts(MoP)方法,通过将问题空间划分为子区域并为每个区域自动构建包含指令和示例的专门专家,解决单一提示覆盖能力不足的问题,在多个基准上平均胜率达81%。

摘要:

大型语言模型(LLMs)在通过语言指令和上下文示例进行提示时,对新任务表现出强大的泛化能力。由于这种能力敏感地依赖于提示的质量,已有多种方法被探索来自动化指令设计。虽然这些方法展示了有前景的结果,但它们也将搜索到的提示限制为单一指令。这种简化显著限制了它们的能力,因为单个无示例的指令可能无法覆盖目标任务整个复杂的问题空间。为了缓解这一问题,我们采用专家混合(Mixture-of-Expert)范式,将问题空间划分为一组子区域;每个子区域由一个专门的专家管理,该专家同时配备一条指令和一组示例。我们开发了一个两阶段过程来为每个区域构建专门的专家:(1)示例分配:受上下文学习与核回归之间理论联系的启发,我们基于语义相似性将示例分组到专家中;(2)指令分配:基于区域的联合搜索为每个专家找到一条指令,以补充分配给该专家的示例,从而产生协同效应。由此得到的方法代号为Mixture-of-Prompts(MoP),在多个主要基准上相对于先前方法取得了81%的平均胜率。

英文摘要:

Large Language Models (LLMs) exhibit strong generalization capabilities to novel tasks when prompted with language instructions and in-context demos. Since this ability sensitively depends on the quality of prompts, various methods have been explored to automate the instruction design. While these methods demonstrated promising results, they also restricted the searched prompt to one instruction. Such simplification significantly limits their capacity, as a single demo-free instruction might not be able to cover the entire complex problem space of the targeted task. To alleviate this issue, we adopt the Mixture-of-Expert paradigm and divide the problem space into a set of sub-regions; Each sub-region is governed by a specialized expert, equipped with both an instruction and a set of demos. A two-phase process is developed to construct the specialized expert for each region: (1) demo assignment: Inspired by the theoretical connection between in-context learning and kernel regression, we group demos into experts based on their semantic similarity; (2) instruction assignment: A region-based joint search of an instruction per expert complements the demos assigned to it, yielding a synergistic effect. The resulting method, codenamed Mixture-of-Prompts (MoP), achieves an average win rate of 81% against prior arts across several major benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑