What the Weight?!：一个面向零样本知识组合的统一框架
What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition
AI总结：
本文提出零样本参数模块组合统一框架，系统比较两种组合方法与五种选择、加权策略，发现集成及简单加权的价值，并揭示 adapter 组合性能具有一定可预测性。
摘要：
模型中封装的知识是决定其在下游任务上最终性能的核心因素。NLP 中的许多研究都聚焦于以高效方法存储和适配不同类型的知识，例如使用专门的模块化结构，以及如何有效地组合这些知识，例如通过学习额外参数。然而，鉴于存在许多可能的选择，目前对这些组合所涉及机制仍缺乏透彻理解，因此仍不清楚应当采用哪些策略。为弥补这一研究空白，我们提出了一个用于零样本模块组合的新颖框架，该框架在单一统一概念下涵盖了选择、加权和组合参数模块的现有方法以及一些新颖变体。围绕领域知识和 adapter 层这一场景，我们的框架对相关概念进行了系统性统一，使我们能够首次对各种零样本知识组合策略开展全面的基准研究。具体而言，我们在广泛的实验设置中测试了两种模块组合方法以及五种选择与加权策略的有效性和效率。结果凸显了集成方法的功效，同时也表明简单但常被忽视的加权方法具有强大潜力。进一步的深入分析使我们理解了加权与 top-k 选择各自的作用，并表明在一定程度上，adapter 组合的性能甚至可以被预测。
英文摘要：
The knowledge encapsulated in a model is the core factor determining its final performance on downstream tasks. Much research in NLP has focused on efficient methods for storing and adapting different types of knowledge, e.g., in dedicated modularized structures, and on how to effectively combine these, e.g., by learning additional parameters. However, given the many possible options, a thorough understanding of the mechanisms involved in these compositions is missing, and hence it remains unclear which strategies to utilize. To address this research gap, we propose a novel framework for zero-shot module composition, which encompasses existing and some novel variations for selecting, weighting, and combining parameter modules under a single unified notion. Focusing on the scenario of domain knowledge and adapter layers, our framework provides a systematic unification of concepts, allowing us to conduct the first comprehensive benchmarking study of various zero-shot knowledge composition strategies. In particular, we test two module combination methods and five selection and weighting strategies for their effectiveness and efficiency in an extensive experimental setup. Our results highlight the efficacy of ensembling but also hint at the power of simple though often-ignored weighting methods. Further in-depth analyses allow us to understand the role of weighting vs. top-k selection, and show that, to a certain extent, the performance of adapter composition can even be predicted.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
