Large language models (LLMs) are highly capable at a variety of tasks given the right prompt, but writing one is still a difficult and tedious process. In this work, we introduce ConstitutionalExperts, a method for learning a prompt consisting of constitutional principles (i.e. rules), given a training dataset. Unlike prior methods that optimize the prompt as a single entity, our method incrementally improves the prompt by surgically editing individual principles. We also show that we can improve overall performance by learning unique prompts for different semantic regions of the training data and using a mixture-of-experts (MoE) architecture to route inputs at inference time. We compare our method to other state of the art prompt-optimization techniques across six benchmark datasets. We also investigate whether MoE improves these other techniques. Our results suggest that ConstitutionalExperts outperforms other prompt optimization techniques by 10.9% (F1) and that mixture-of-experts improves all techniques, suggesting its broad applicability.
One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Prompts
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Korea Advanced Institute of Science & Technology（韩国科学技术院）；Penn State University（宾夕法尼亚州立大学）；University of Maryland, College Park（马里兰大学学院公园分校）
Few-Shot Anaphora Resolution in Scientific Protocols via Mixtures of In-Context Experts
School of Interactive Computing（交互计算学院）；Georgia Institute of Technology（佐治亚理工学院）
指代消解是跨语言、文本类型和领域的信息抽取的重要任务，激发了对不需要大的注释数据集的方法的需求。上下文学习是一种很有前途的方法，但将上下文学习应用于指代消解还存在一些挑战。例如，编码一个由以下内容组成的上下文演示：一个照应词、一个段落长度的上下文和一个相应的先行词列表，需要在一个长序列的标记上调节语言模型，限制每个提示的演示数量。在本文中，我们提出了MICE（Mixtures of In-Context Experts），我们证明了它对于科学协议中的Few-Shot回指解析是有效的（Tamari et al.，2021年）。在只给出少量训练样本的情况下，MICE结合了数百名上下文专家的预测，使F1分数比竞争性即时检索基线提高了30%。此外，我们表明MICE可以用于训练紧凑的学生模型，而不牺牲性能。据我们所知，这是第一个工作，目前的实验结果表明，在上下文学习的有效性的任务，Few-Shot照应决议在科学协议。
CoCoA-Mix: Confusion-and-Confidence-Aware Mixture Model for Context Optimization
Urban Robotics Lab, School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Republic of Korea（乌尔班机器人实验室，电气工程学院，韩国科学技术院，大韩民国）
Learning Perceptual Geometry of Natural Odor Mixtures from LLM-Derived Descriptors
Center for Brain Science, Harvard University（哈佛大学脑科学中心）；Department of Molecular and Cellular Biology, Harvard University（哈佛大学分子与细胞生物学系）；Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University（哈佛大学肯普ner自然与人工智能研究所）
An automatically discovered chain-of-thought prompt generalizes to novel models and datasets
Medical University of Vienna（维也纳医科大学）；Johannes Kepler University Linz（林茨约翰内斯·开普勒大学）；KTH Royal Institute of Technology（KTH皇家理工学院）；Humboldt University Berlin（柏林洪堡大学）
MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning
iFLYTEK Co., Ltd（科大讯飞股份有限公司）；University of Pennsylvania（宾夕法尼亚大学）；Lenovo Connect Co., Ltd（联想智联有限公司）；Niuxin Network Technology Co., Ltd（牛信网络科技有限公司）；East China Normal University（华东师范大学）
University of Michigan（密歇根大学）；UC Riverside（加州大学河滨分校）；Google Research（谷歌研究院）；University of Southern California（南加州大学）；University of British Columbia（不列颠哥伦比亚大学）
Evolutionary Soups: Evolving Mixture-of-Experts for Multi-Objective LLM Alignment
Fraunhofer Institute for Applied Information Technology FIT（弗劳恩霍夫应用信息技术研究所FIT）；University of Cologne（科隆大学）；University of Stuttgart（斯图加特大学）；University of Southampton（南安普顿大学）；Soochow University（苏州大学）；University Hospital of Cologne（科隆大学医院）
Enhancing Vision-Language Models for Autonomous Driving through Task-Specific Prompting and Spatial Reasoning
University of Chinese Academy of Sciences（中国科学院大学）；Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences（中国科学院空间利用技术与工程中心）
Decomposing Refusal Steering in Mixture-of-Experts Models
Department of Interdisciplinary Studies（交叉学科研究部）；University of Montana（蒙大拿大学）；Department of Pharmacy Practice & Science（药学与科学系）；University of Arizona（亚利桑那大学）；European Bioinformatics Institute（欧洲生物信息研究所）；European Molecular Biology Laboratory（欧洲分子生物学实验室）；Wellcome Genome Campus（沃氏基因组校园）
MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper
Rochester Institute of Technology（罗切斯特理工学院）；University of Rochester（罗切斯特大学）；Meta AI；Rice University（莱斯大学）；Case Western Reserve University（凯斯西储大学）；U.S. DEVCOM Army Research Laboratory（美国陆军作战能力发展司令部陆军研究实验室）；University of Missouri - Kansas City（密苏里大学堪萨斯城分校）；UC Davis（加州大学戴维斯分校）
MoPD: Mixture-of-Prompts Distillation for Vision-Language Models
Department of Computer Science and Engineering, Southern University of Science and Technology（计算机科学与工程系，南方科技大学）；School of Mathematical and Computer Sciences, University of Adelaide（数学与计算机科学学院，阿德莱德大学）
MoPE: Mixture of Prompt Experts for Parameter-Efficient and Scalable Multimodal Fusion
Department of Computing, The Hong Kong Polytechnic University（计算系，香港理工大学）；Research Institute of Multiple Agents and Embodied Intelligence, Pengcheng Laboratory（多智能体与具身智能研究院，鹏城实验室）
Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding
Peking University（北京大学）；National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；School of Computer Science, Peking University（北京大学计算机学院）
